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ABSTRACT 


With  the  acceptance  of  Unmanned  Aerial  Vehicles  (UAVs)  as  a  primary  platform 
within  the  Department  of  Defense  (DOD)  for  gathering  intelligence  data,  the  amount  of 
video  information  being  recorded,  analyzed,  and  archived  continues  to  grow. 
Mechanisms  for  quickly  locating  and  retrieving  video  segments  of  interest  amongst  the 
many  hours  of  recorded  video  are  required  to  accommodate  the  rapid  turnaround 
expected  in  today’s  wartime  planning  environments.  This  research  demonstrates  that 
text-based  data  accompanying  UAV  video  yields  sufficient  information  to  identify  and 
create  data  items  that  can  be  indexed  to  provide  for  rapid  identification  and  retrieval  of 
video  segments  of  interest.  Four  attributes  are  derived  or  calculated  from  mission-related 
telemetry  and  target  data:  look-direction,  look-distance,  zoom,  and  solar  illumination 
direction.  These  attributes  provide  indicators  of  potential  scene  content  and  image 
quality  thus  allowing  analysts  to  select  the  best  (clearest,  most  detailed)  images  of  a 
target  or  target  area.  A  relational  database  and  two  tools,  a  query  and  profile  tool,  are 
implemented  to  store  the  data  and  provide  for  the  retrieval  and  presentation  of  video 
segments  to  the  user.  An  analysis  of  the  results  shows  the  methodology  to  be  favorable 
for  locating  and  retrieving  video  segments  of  interest.  Several  recommendations  to 
enhance  the  methodology  are  provided  at  the  conclusion  of  the  paper. 


EXTRACTING  MISSION  SEMANTICS  FROM 


UNMANNED  AERIAL  VEHICLE  TELEMETRY  AND 

FLIGHT  PLANS 

1  INTRODUCTION 


Battlefield  superiority  today  can  be  significantly  influenced  by  the  availability  of 
real-time  or  near-real-time  information.  Commanders  can  no  longer  afford  long  delays  in 
gathering,  processing,  and  disseminating  intelligence  data;  they  require  immediate  and 
continuous  information  flow  to  support  a  dynamic  battlefield.  The  Joint  Requirements 
Oversight  Council  recognized  the  changing  nature  of  warfare  and  issued  the  following 
mission  need  statement: 

The  warfighting  Comanders-in-Chief  (CINCs)  have  a  need  to  provide 
commanders  a  responsive  capability  to  conduct  wide-area,  near-real-time 
reconnaissance,  surveillance,  and  target  acquisition  (RSTA),  command  and 
control,  signals  intelligence  (SIGINT),  electronic  warfare  (EW),  and  special 
operations  missions  during  peacetime  and  all  levels  of  war  against 
defended/ denied  areas  over  extended  periods  of  time.  The  evolution  of  the 
hostile  surface-to-air  and  air-to-air  threat  and  their  collective  effectiveness 
against  manned  aircraft  and  satellites  can  generate  unacceptably  high  attrition 
rates.  Current  systems  cannot  perform  these  missions  in  a  timely,  responsive 
manner  in  an  integrated  hostile  air  defense  environment  without  high  risk  to 
personnel  and  costly  systems.  There  is  a  need  for  a  capability  which  can  be 
employed  in  areas  where  enemy  air  defenses  have  not  been  adequately 
suppressed,  in  heavily  defended  areas,  in  open  ocean  environments,  and  in 
contaminated  environments.  Nuclear  survivability  is  required  as  necessary  to 
perform  missions  in  a  nuclear  contaminated  environment,  including  operating 
in  the  presence  of  high-altitude  EMP  [ACCC098]. 
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One  of  the  ways  the  Defense  Department  is  addressing  the  changing  face  of 
warfare  is  by  employing  Unmanned  Aerial  Vehicles  (UAVs)  in  addition  to,  or  in  lieu  of, 
traditional  manned  and  satellite  systems.  Manned  reconnaissance  systems  are  associated 
with  high  operating  costs  and  every  manned  mission  places  human  life  at  risk.  Satellite 
assets  are  national  assets  and  may  not  be  visible  when  the  commander  in  the  field  needs 
them.  In  some  instances,  satellite  assets  must  be  moved  into  position  before  they  are 
available  to  receiving  stations,  and  orbiting  satellites  suffer  from  limited  exposure  at 
discrete  intervals.  Satellites  also  suffer  from  potentially  long  delays  in  processing  and 
disseminating  time-critical  information  to  supported  commanders. 

UAVs,  on  the  other  hand,  are  ideal  platforms  when  near-real -time  information  is 
required  and  when  operating  in  high-threat  or  heavily  defended  environments  where  the 
potential  for  loss  to  high-value  manned  systems  is  possible.  UAVs  can  penetrate  enemy 
air  space  with  a  low  probability  of  detection  and  loiter  over  a  target  for  extended  periods 
of  time,  day  or  night,  recording  imagery  in  visual,  infrared,  and  numerous  electro-optical 
bands.  Their  maneuverability  and  rapid  deployment  capability  means  they  can  support 
short-notice,  ad-hoc  planning  scenarios. 

UAVs  currently  in  operation  in  the  Department  of  Defense  (DOD)  are  designed  to 
perform  either  tactical  or  strategic  intelligence  missions  [CONGR98].  Tactical 
intelligence  provides  the  brigade  commander  with  the  strength,  quality,  and  position  of 
enemy  forces  just  over-the-hill  (OTH).  The  Army’s  Hunter  platform  was  developed  to 
provide  this  capability.  Strategic  intelligence  refers  to  longer-range  information  gathered 
by  reconnaissance  satellites  or  aircraft  or  long-range,  endurance  UAV  platforms  such  as 
the  Air  Force’s  Predator. 
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To  accomplish  these  varied  missions,  UAVs  are  equipped  with  several  types  of 
sensor  systems  including  those  capable  of  providing  color  and  black  and  white  video  feed 
as  well  as  still  images.  Additionally,  full  motion  video  is  available  in  visible,  electro- 
optical,  and  infrared  formats.  Given  the  UAV’s  ability  for  long  loiter  times,  there  is 
potential  for  large  quantities  of  video  footage  to  be  accumulated  in  a  single  mission. 

With  these  large  quantities  of  data  comes  the  problem  of  trying  to  locate  video  segments 
of  interest  amongst  the  many  hours  of  footage. 

1.1  Problem 

With  the  advent  and  proliferation  of  inexpensive  computer  storage  mechanisms, 
storing  video  footage  is  only  a  small  problem.  A  larger  challenge  is  involved  in 
retrieving  specific  segments  of  archived  footage.  Currently,  intelligence  analysts  must 
painstakingly  review  video  in  a  serial  fashion.  Much  research  is  being  done,  however,  to 
improve  the  ability  to  retrieve  specific  video  segments  from  a  larger  sequence.  The  most 
promising  areas  are  those  which  are  focusing  on  the  semantic  content  of  the  video  to 
identify  scenes  or  segments  of  interest.  Unfortunately,  the  footage  being  used  to  further 
this  research  often  involves  segments  where  visible  and  distinguishable  motion  of 
elements  within  the  video  is  taking  place  or  where  apparent  changes  in  scene  content 
occur.  UAV  video,  on  the  other  hand,  most  often  involves  segments  where  no  motion 
(other  than  the  camera  motion  itself)  or  scene  content  change  is  taking  place.  These 
segments  are  uninteresting  for  commercial  purposes  but  may  contain  targets  of  high 
interest  to  intelligence  analysts.  To  identify  and  index  these  segments,  other  means  must 
be  employed  and  other  sources  of  indexable  metadata  must  be  discovered. 
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1.2  Scope 

The  key  to  a  robust  video  retrieval  system  is  in  its  ability  to  provide  access  to 
specific  scenes  or  segments  of  interest  to  the  user.  The  more  metadata  the  user  has 
available  to  correlate  with  scene  or  operational  semantics,  the  more  capability  they  will 
have  for  effective  ad-hoc  querying  of  the  video  stream.  The  commercial  market  is 
primarily  focusing  on  the  content  of  the  video  footage  itself  to  produce  indexes  for 
retrieval.  This  research  will  focus  on  supplemental  data  that  complements  and  further 
defines  the  content  of  the  video.  Telemetry  data  gathered  during  the  mission,  target  lists 
produced  during  mission  planning,  and  even  solar  position  information  can  provide 
useful  clues  to  both  the  video  content  and  the  potential  quality  of  the  images  in  the  video. 
The  quantity  of  this  data,  however,  can  be  very  large  (many  millions  of  bytes). 

Therefore,  it  is  the  intention  of  this  research  to  identify  portions  of  this  information  that 
will  offer  clues  to  scene  content  and  flight-pattern  semantics  that  will  assist  users  in 
locating  and  accessing  video  segments  of  interest.  It  is  proposed  that  this  information, 
when  combined  with  semantic  information  gleaned  directly  from  the  video  footage,  will 
offer  a  more  complete  mechanism  for  scene  or  segment  identification  and  subsequent 
retrieval  from  a  video  database  management  system  (VDBMS). 
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1.3  Objectives 

This  research  demonstrates  that  text-based  data  accompanying  UAV  video  yields 
sufficient  information  to  identify  and  create  data  items  that  can  be  indexed  to  provide  for 
the  identification  and  retrieval  of  video  segments  of  interest.  Four  phases  have  been 
identified  for  accomplishing  this  goal.  Supporting  objectives,  as  well  as  the  phase  in 
which  they  are  completed  are  provided  in  the  following  table. 


Table  1 :  Research  Objectives  and  Phases 


1 .  Collect  and  analyze  telemetry  and  target  data  to  identify  elements  that  will  yield 
potentially  useful  data  items  and  indexes  into  the  video  stream. 

Phase  I 

2.  Design  algorithms  for  extracting  and  deriving  information  from  the  telemetry  stream 
based  on  target  location  and  data  items  identified  in  Objective  1. 

Phase  II 

3.  Design  algorithms  to  calculate  the  angle  and  azimuth  of  the  sun  to  provide  clues  to 
potential  image  quality  and  offer  a  more  precise  querying  capability. 

Phase  II 

4.  Design  and  implement  a  relational  database  to  store  data  items  identified  in  Objective  1 
and  solar  information  in  Objective  3. 

Phase  II 

5.  Design  and  implement  a  user  interface  to  provide  querying  of  data  (based  upon  the  data 
items  identified  in  Objective  1)  and  retrieval  of  video  segments  containing  images  of 
targets  identified  in  target  lists. 

Phase  III 

6.  Design  and  implement  a  mission  profiler  based  on  telemetry  and  target  data  that  provides 
for  the  retrieval  of  video  segments. 

Phase  IV 

1.4  Assumptions 

A  few  assumptions  were  made  prior  to  proceeding  with  this  research.  Firstly,  it 
has  been  assumed  the  information  required  for  investigation  would  be  made  available 
upon  request  from  the  sponsor.  Sufficient  data  sets  were  made  available  for  this  research. 
Secondly,  it  has  been  assumed  that  accompanying  the  data  will  be  valid  definitions  that 
provide  insight  into  the  semantic  meaning  of  individual  data  items.  This  was  available 
for  telemetry  data  but  not  for  target  information.  Lastly,  it  has  been  assumed  the  data 
provided  by  the  sponsor  will  remain  unclassified  and  be  a  good  representation  of  actual 
UAV  missions.  All  video  and  telemetry  data  satisfied  this  assumption. 
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1.5  Thesis  Organization 

There  are  5  chapters  in  this  document.  Chapter  1  introduces  the  need  for  random 
access  into  UAV  video  to  expedite  analysis  of  the  information.  Accompanying  text- 
based  data  is  offered  as  a  source  for  developing  indexes  into  the  video  stream  to  expedite 
identification  and  retrieval  of  segments  of  interest.  Chapter  2  offers  a  brief  overview  of 
how  UAVs  are  being  employed  in  military  as  well  as  commercial  and  government 
sectors.  Also  covered  are  unique  attributes  of  national  security  applications  and  a  model 
for  representing  metadata  associated  with  video  footage.  Chapter  3  outlines  the 
methodology  employed  by  this  research  to  prepare  data  and  identify  or  calculate  the 
metadata  to  be  associated  with  the  video.  A  discussion  of  the  requirements  for  the 
interface  used  during  the  demonstration  phase  of  this  research  is  also  provided  in  Chapter 
3.  Chapter  4  presents  the  results  of  this  research  and  details  the  interface  used  to  present 
indexed  metadata  to  the  user.  This  chapter  demonstrates  that  random  access  into  UAV 
video  based  on  indexed  metadata  was  achieved  as  proposed.  Chapter  5  offers 
recommendations  for  future  research  and  offers  suggestions  as  to  how  recent 
technological  advances  and  information  visualization  techniques  can  enhance  the  utility 
of  the  methodology.  Finally,  Appendix  A  contains  directions  for  obtaining  copies  of  the 
data  and  software  developed  during  this  research. 
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2  BACKGROUND 


Unmanned  Aerial  Vehicle  systems  have  been  in  use  for  tactical  reconnaissance 
since  the  Vietnam  War  [CONGR98];  however,  broad  military  and  commercial 
application  has  taken  place  only  over  the  past  two  decades.  As  a  result  of  their  increased 
use,  new  methods  for  storing  and  exploiting  the  vast  amounts  of  information  being 
gathered  have  become  desirable.  Prior  to  investigating  storage  and  retrieval  issues 
specifically,  a  discussion  of  several  uses  of  UAVs  is  provided  for  background.  This  will 
provide  an  understanding  of  the  types  and  magnitude  of  data  that  must  be  accommodated 
by  visual  information  retrieval  (VIR)  systems.  This  chapter  will  introduce  some  of  the 
government,  commercial,  and  military  applications  of  UAVs  and  how  military 
applications  differ  from  commercial.  A  brief  introduction  to  VIR  systems  is  included  as 
well  as  how  military  content  based  recognition  (CBR)  systems  must,  by  nature,  differ 
from  commercial.  Following  the  discussion  of  CBR  systems  is  an  introduction  to  a 
prototype  VIR  system  in  that  utilizes  the  model  of  information  storage  and  retrieval  this 
research  extends.  This  chapter  concludes  with  an  introduction  to  digital  versatile  disk 
(DVD)  technology  since  this  technology  will  be  recommended  as  the  future  storage 
mechanism  for  this  research. 

2.1  Applications  of  UAVs 

Conventional,  manned  aircraft  are  encumbered  with  several  limiting  factors 
associated  with  cost,  design,  and  safety.  The  expense  of  purchasing,  maintaining,  and 
flying  manned  aircraft  often  make  them  cost-prohibitive  for  broad  commercial  and 
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military  use.  Additionally,  design  limitations  (such  as  fuel  capacity)  equate  to  a  limited 
time-on-target  capability,  and,  as  in  the  case  of  military  applications,  loitering  around  the 
target  may  pose  hazardous  to  both  man  and  machine. 

UAVs,  however,  offer  a  low-cost,  safe  alternative  to  manned  vehicles. 
Government,  military,  and  commercial  sectors  are  exploring  and  embracing  the  ever- 
increasing  applications  of  unmanned  aircraft.  A  few  of  the  applications  are  discussed 
below. 


2.1.1  Government  and  Commercial 

UAVs  are  being  utilized  by  government  entities  such  as  the  Department  of  Energy 
(DOE)  and  the  Central  Intelligence  Agency  (CIA).  UAVs  are  also  finding  application  in 
aerial  cropland  survey  and  monitoring  forest  fires.  Likewise,  they  are  being  proposed  for 
farming  and  environmental  research  and  monitoring.  More  specific  details  follow. 

DOE.  The  DOE  has  the  responsibility  of  cleaning  and  disposing  of  hazardous 
contaminants  resulting  from  nuclear  materials  production  [ALBER96].  UAVs  are  being 
utilized  to  assist  in  the  characterization  and  monitoring  of  waste  sites  ranging  in  size  from 
less  than  50  acres  to  over  10,000  acres  [PENDE96]  [NYQUI96].  Manned  airborne 
systems  have  proven  cost  prohibitive  for  small  sites.  Additionally,  long  lead-times  for 
flight  planning  make  them  unsuitable  for  emergency  response  scenarios.  UAVs  offer  a 
cost-effective  alternative  for  surveying  small  sites  (<  50  acres)  which  comprise  most  of 
the  sites  managed  by  the  DOE.  Hobby-class  systems  fitted  with  relatively  inexpensive 
photographic  equipment  are  used  to  supplement  or  replace  manned  systems.  Figure  1 
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shows  the  reduced  operating-costs  that  have  motivated  the  move  to  small,  inexpensive 
platforms  away  from  larger,  more  expensive  manned  ones. 


Figure  1:  Cost  per  Acre  to  Site  Size  Comparison  for  Geophysical  Surveys  [ALBER96] 

CIA.  The  CIA  considers  the  I-Gnat  UA V  as  one  of  its  key  intelligence-gathering 
platforms  [VIZAR99J.  I-Gnat  was  used  successfully  to  monitor  Serbian  troops  during 
the  1999  war  in  Kosovo.  It  was  also  used  in  1997  to  monitor  the  terrorist  take-over  of  the 
Japanese  embassy  in  Peru.  Recent  upgrades  in  the  onboard  Synthetic  Aperture  Radar 
(SAR)  provide  unprecedented  image  clarity  for  surveillance  and  reconnaissance 
applications. 

Other  Uses.  Commercial  applications  of  UAV  technology  are  many  and  varied. 
Companies  performing  cropland  surveys  will  benefit  from  the  low  cost  and  high-quality 
images  available  with  low  and  slow  flying  unmanned  aircraft  [HOVER99].  Other  farm- 
related  uses  being  proposed  include  livestock  and  crop  health  monitoring  where  the  costs 
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or  operating  environments  are  not  conducive  to  employing  manned  aircraft.  Possible 
environmental  applications  include  monitoring  of  coastal  fishing  zones,  unobtrusive 
wildlife  surveys,  volcano  monitoring,  and  tornado  spotting  and  tracking  [STILW95]. 

2.1.2  Military 

Military  UAV  systems  are  designed  to  fulfill  a  broad  range  of  operational 
reconnaissance  and  surveillance  objectives.  Platform  categories  include  short-range 
(such  as  Hunter),  medium-range  endurance  (such  as  Predator),  and  high-altitude 
endurance  (such  as  Darkstar).  Brief  introductions  to  a  few  of  their  many  roles  are 
provided  below. 

Near-Real-Time  (NRT)  Targeting.  UAVs  can  provide  full-time  coverage  of  an 
area  and  provide  near-real-time  intelligence  as  to  target  location  and  behavior. 

Battle  Damage  Assessment.  UAVs  can  locate  targets,  loiter  in  the  area  and 
provide  post-strike  assessment.  The  near  immediate  feedback  assists  commanders  in 
determining  if  a  second  strike  is  required  on  a  target. 

Special  Operations.  UAVs  can  be  used  to  support  tracking  of  high-interest 
individuals  or  organizations.  Additionally,  a  UAV’s  ability  to  look  over  the  hill  (OTH) 
with  a  downlink  to  forces  in  the  field  can  assist  safe  ingress  and  egress  of  forces  into 
hostile  territory. 

Blockade  and  Quarantine  Enforcement.  UAVs  provide  a  cost-effective  means 
to  monitor  economic,  military,  and  drug-enforcement  blockades  and  quarantines. 
Enforcement  personnel  are  then  free  to  perform  other  missions. 
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Sensitive  Reconnaissance  Operations  (SRO).  Missions  in  this  category  by 
nature  pose  a  risk  to  personnel  safety  and/or  are  politically  sensitive.  UAVs  offer  a  safe, 
effective  means  of  carrying  out  these  types  of  missions. 

United  Nations  (UN)  Treaty  Monitoring.  Low  operating  cost  and  long  time-on- 
target  capabilities  make  UAVs  ideal  for  monitoring  compliance  with  UN  resolutions  and 
treaties.  Adherence  can  be  monitored  without  introducing  UN  representatives  into 
potentially  hostile  areas  and  conditions. 

Counter  Drug.  Endurance  UAVs  can  assist  authorities  in  identifying,  tracking, 
and  apprehending  trafficking  outlaws. 

Communications.  UAVs  have  the  potential  to  extend  the  range  of 
communications  systems  by  acting  as  a  relay.  This  capability  could  help  bridge  the  gap 
between  front-line  forces  and  rear-echelon  commanders  monitoring  battlefield  progress 
and  performing  dynamic  mission  planning.  This  capability  could  also  substitute  for 
overwhelmed  or  damaged  satellite  links. 

Table  2  summarizes  the  various  uses  of  UAVs  within  commercial,  government, 
and  military  communities.  This  list  is  not  intended  to  be  all-inclusive. 

Table  2  :  Commercial,  Government,  and  Military  Uses  of  UAVs 


Commercial  &  Government 

Military 

Characterization  &  monitoring  of  waste  sites 

Near-real-time  targeting 

Waste  site  surveys 

Battle  damage  assessment 

Surveillance  &  reconnaissance 

Special  operations 

Monitoring  crop  &  herd  health 

Blockade  &  quarantine 
enforcement 

Monitoring  coastal  fishing  zones 

Sensitive  reconnaissance  operations 

Wildlife  surveys 

United  Nations  treaty  monitoring 

Monitoring  volcanic  activity 

Counter  drug 

Storm  tracking  &  monitoring 

Communications 
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2.2  Uniqueness  of  Military  Application 

While  UAVs  are  equally  adept  at  supporting  military,  government,  and 
commercial  applications,  the  modus  operandi  and  payload  usage  often  differ  in  three 
areas:  operational  altitude,  proximity  and  angle  of  approach  to  the  target  area,  and  content 
and  use  of  payload  output.  These  factors  must  be  considered  in  the  development  or 
acquisition  of  appropriate  video  storage  and  retrieval  systems. 

The  commercial  applications  described  above  can  be  accomplished  while 
operating  the  platform  at  any  altitude  allowed  by  the  terrain  and  government  regulations. 
Military  UAVs,  however,  are  often  operated  in  hostile  territory.  In  these  instances,  a 
minimum  altitude  may  be  imposed  on  the  platform  operator  to  avoid  detection  by  the 
enemy.  These  circumstances  may  serve  to  reduce  the  clarity  or  detail  in  the  video  images 
being  recorded. 

Commercial  applications  are  further  supported  by  more  freedom  in  the  angle  of 
approach  and  view  of  the  target  area.  In  many  cases,  the  platform  can  be  flown  directly 
over  the  target  area  at  low  altitudes.  A  vertical  view-angle  and  low  operating  ceiling 
support  high  quality,  highly  detailed  video  and  images.  For  the  same  reasons  discussed 
in  the  previous  paragraph,  military  platforms  are  restricted  in  many  instances  to  an 
oblique  view  of  the  target  at  distances  of  several  kilometers.  The  images  and  video 
produced  by  these  less  than  optimal  operating  parameters  make  analysis  of  the  output 
more  difficult. 

Finally,  commercial  applications  have  different  requirements  for  storing  and 
retrieving  payload  output.  The  amount  of  output  (video,  imagery,  text-based)  produced 
during  military  operations  and  the  fact  that  most  of  the  output  is  archived  for  post- 


12 


mission  analysis  makes  storage  and  retrieval  a  prime  consideration  during  selection  of  an 
appropriate  storage  and  retrieval  system.  Most  commercial  applications,  on  the  other 
hand,  do  not  approach  the  quantity  of  output  produced  by  the  military.  Additionally, 
military  analysts  are  often  pressed  to  analyze  and  produce  reports  for  recently  executed 
missions.  This  compressed  schedule  and  sense  of  urgency  requires  a  degree  of  rapid  and 
precise  random  access  into  the  video  stream  not  usually  required  of  commercial 
applications.  Table  3  summarizes  the  difference  between  the  use  of  UAVs  for 
commercial  and  military  applications. 


Table  3:  Comparison  of  Commercial  and  Military  UAV  Applications 


Commercial 

Military 

Operate  at  any  altitude  allowed  by  the  terrain 
and  government  regulations  allowing  low-level 
passes  over  the  target  producing  highly  detailed 
video  images 

Operations  over  and  around  hostile  territory 
may  impose  a  minimum  operating  altitude  thus 
reducing  the  detail  and  clarity  of  video  images 

Approach  the  target  from  any  angle  allowed  by 
terrain  and  regulations;  can  pass  directly  over 
the  target  at  low  altitudes  to  obtain  the  best 
video  images 

Hostile  operating  environments  restrict  angle  of 
approach  and  minimum  approachable  range  to 
the  target  thus  reducing  detail  and  clarity  of 
video  images 

Lower  storage  space  requirements 

High  storage  space  requirement 

Less  urgency  in  analyzing  output 

Rapid  processing  and  analysis  of  output  is 
required 

2.3  Video  Information  Retrieval  Systems 

The  increased  availability  and  broad  use  of  video  information  has  fueled  interest 
in  the  area  of  video  information  retrieval  (VIR).  VIR  systems  seek  to  retrieve  video 
information  based  on  user-relevant  queries.  They  seek  to  provide  the  ability  to  query  and 
retrieve  video  information  as  easily  as  text-based  information. 
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Accessing  information  that  describes  the  content  of  the  video  is  an  essential 
element  of  VIR  systems  [DELB  199].  Data  that  describes  the  visual  elements  of  the  video 
(such  as  color,  texture,  shape,  and  spatial  relationships),  editing  effects  (such  as  pans, 
cuts,  and  camera-angles),  and  object  motion  provides  indexes  for  quickly  retrieving 
relevant  video  segments. 

Through  surveys  of  multimedia  users,  [ROWE94]  characterized  the  types  of 
video  queries  which  should  be  supported  by  VIR  systems.  Based  on  their  findings,  they 
recommend  the  following  indexes  to  satisfy  user  queries: 

Bibliographic  data  or  content-independent  metadata  includes  information  about 
the  entire  video  [DELBI99].  Candidate  attributes  include  title,  subject,  creator,  date  of 
creation,  etc.  The  bibliographic  elements  that  could  be  associated  with  video  used  in  this 
research  include  platform  type,  mission  date,  mission  time,  etc. 

Structure  data  includes  attributes  that  describe  the  hierarchy  of  the  video. 
Segment,  scene,  and  shot  are  terms  commonly  used  to  refer  to  increasingly  decomposed 
elements  of  the  video.  The  lowest  level,  the  shot,  describes  the  continuous  action 
between  the  start  and  stop  of  camera  operation.  During  the  collection  of  UAV  video,  the 
camera  operation  is  continuous.  Therefore,  segments  and  scenes  are  the  only  levels  of 
decomposition  referred  to  in  this  research  (Figure  2). 
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Scene  A.l  Scene  A.2 


Figure  2:  Video  Structure 


Content  data,  also  referred  to  as  content-descriptive  metadata,  reveals 
information  about  the  semantic  content  of  the  video  [DELB  199].  These  attributes  are 
reflected  by  both  static  elements,  such  as  objects  or  people,  and  dynamic  elements,  such 
as  movement  of  objects. 

Effective  video  retrieval  systems  seek  to  utilize  all  available  information  sources, 
including  closed-captioned  text,  audio,  manual  annotations,  and  visual  information  to 
build  a  list  of  attributes  (metadata)  describing  the  video  [BOLLE98].  Ideally,  video  clips 
should  be  retrieved  from  larger  sequences  based  on  both  temporal  structure  as  well  a 
semantic  content.  The  latter  goal,  referred  to  as  content-based  retrieval  (CBR),  poses  the 
most  challenge  to  VIR  developers  and  is  the  focus  of  much  research  including 
[GRIFF96],  [PEREZ96],  [POPE98],  [BOLLE98],  and  [BRONE97].  Approaches  range 
from  using  visual  attributes  such  as  color,  shape,  and  texture,  as  in  the  Query  By  Image 
Content  (QBIC)  project  [NIBLA93],  to  indexing  associated  audio  [CHAN96]. 

Visual  information  is  accessed  most  effectively  when  users  are  given  the  ability  to 
switch  between  querying  and  browsing  available  information.  Browsing  offers  users  a 
panoramic  view  of  the  data  and  is  useful  for  quickly  locating  and  isolating  information  of 
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interest.  Integral  to  effective  browsing  is  the  use  of  effective  visualization  techniques. 
Tools  should  be  designed  to  present  visual  information  to  the  user  in  a  manner  best 
understood  by  the  user  and  which  summarizes  information  most  effectively.  Techniques 
being  explored  include  mosaics,  thumbnail  images,  and  representative  points  in  multi¬ 
dimensional  space. 

2.4  CBR  Systems  for  National  Security  Applications 

Research  and  development  of  content  based  recognition  (CBR)  systems  for 
commercial  applications  continue  to  receive  increased  attention.  However,  applications 
in  the  national  security  sector  have  received  less  attention  according  to  [BRONE97]. 
Defense  departments,  intelligence  organizations,  and  law  enforcement  agencies  are 
collecting  video  information  at  an  ever-increasing  rate.  Due  to  the  different  types  of  data 
gathered  and  the  varying  methods  of  collection,  these  organizations  require  CBR  systems 
with  different  characteristics  than  commercial  systems.  Some  of  the  characteristic 
differences  include  [BRONE97]: 

Greater  variety  of  media  types.  Visual  (electro-optical)  and  non-visual 
(infrared,  synthetic  aperture  radar  (SAR),  and  multispectral)  video  and  images  are 
collected  on  a  regular  basis  by  national  security  agencies.  CBR  systems  must  be  able  to 
support  the  representation  of  content  in  both  visual  and  non-visual  forms. 

Little  internal  structure.  Commercial  video  applications  often  employ 
deliberate  use  of  camera  angles,  lighting,  and  motion.  These  elements  provide  internal 
structure  that  can  be  exploited  by  CBR  systems  to  identify  probable  scene  content.  A 
reconnaissance  or  surveillance  video,  on  the  other  hand,  is  free  flowing  and  presents  little 
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or  no  structure.  Additionally,  well  defined  colors,  textures,  and  shapes  are  a  general  rule 
in  commercial  video.  National-security  applications,  however,  have  such  a  broad  range 
of  subject  matter  that  these  attributes  are  not  useful  discriminators  for  identify  the  content 
of  a  video  scene. 

Large  amounts  of  metadata.  National  security  applications  typically  provide 
much  more  metadata  than  is  commonly  associated  with  commercial  video.  Sensor 
information  such  as  look-angle,  field-of-view,  and  image  band  as  well  as  precise  time  and 
position  measurements  are  commonly  associated  with  this  type  of  video.  This  metadata 
should  be  exploited  by  CBR  systems  for  this  domain  because  it  provides  clues  to  the  type 
and  content  of  the  video. 

Consideration  of  multiple  data  sources.  Often,  overlapping  data  sources  are 
available  which  should  be  correlated  and  considered  during  the  query  process.  Satellite 
imagery,  full-motion  video,  audio  transmissions,  environmental  data,  and  intelligence 
information  (Human  Intelligence  (HUMINT),  Signal  Intelligence  (SIGINT),  etc.) 
combine  to  provide  the  composite  picture  required  by  analysts  of  national  security  data. 

Table  4  summarizes  the  differences  between  commercial  and  national  security 
applications.  As  stated  above,  these  factors  must  be  considered  during  the  development 
of  CBR  systems  that  will  be  utilized  by  national  security  agencies. 


Table  4:  Differences  Between  Commercial  and  National  Security  Applications 


Characteristic 

Commercial  Systems 

National  Security  Systems 

Media  Type 

visual 

visual,  non-visual 

Degree  of  Internal  Structure 
of  Video  Elements 

high 

low 

Amount  of  Metadata 

small 

large 

Data  Sources 

few 

many 
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While  CBR  systems  remain  a  topic  of  research,  the  DOD  has  advanced  its  efforts 
to  obtain  systems  that  will  enhance  the  ability  to  retrieve,  analyze,  and  make  decisions 
upon  the  vast  amounts  video  data  available.  One  such  effort  has  been  incorporated  in  the 
Multi-Source  Integration  and  Intelligence  Analysis  (MSIIA)  prototype  system  developed 
by  the  MITRE  Corporation. 

2.5  MSIIA 

MSIIA  provides  a  consolidated  view  of  an  area  of  interest  by  integrating  multiple 
data  sources  and  media  types  into  one  system.  By  integrating  multiple  sources  of 
information,  analysts  are  able  to  determine  the  sensor  coverage  for  a  given  time  and 
space.  One  source  of  information  is  full-motion  video  collected  by  UAV  platforms.  The 
methodology  used  by  MITRE  to  represent  metadata  associated  with  video  footage  is  of 
particular  interest  and  the  reason  it  is  presented  as  background  information  in  this  thesis. 
Brief  discussions  of  the  methodology  and  benefits  follow;  the  reader  is  directed  to 
[HANSE98]  for  an  in-depth  discussion. 

2.5.1  Full-Motion  Video  Track  Generation 

Support  information  available  from  ground  stations  or  on-board  systems  makes 
possible  the  construction  of  tracks  representing  the  UAV  flight  path,  sensor  aim-points, 
and  a  general  prediction  of  the  field-of-view  at  any  given  time  (Figure  3).  At  a  minimum 
the  information  available  includes: 
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Xv :  the  longitude  of  the  platform 
Yv :  the  latitude  of  the  platform 
H  :  the  altitude  of  the  platform 
Xs:  the  longitude  of  the  sensor 

Ys:  the  latitude  of  the  sensor 

Ds:  the  distance  from  the  platform  to  the  sensor  aim- 

point 

T  :  the  time  the  video  was  captured 

Ws:  the  width  of  the  field-of-view  of  the  sensor 


Figure  3:  Track  Attributes  [HANSE98] 

Multiple  arcs  connecting  discrete  data  capture  points  (or  nodes)  represent  a  track. 
Each  node  contains  the  metadata  associated  with  a  discrete  capture  point.  Since  each 
node  contains  a  time  stamp,  it  is  possible  to  determine  how  far  into  a  given  track  a 
particular  data  point  lies.  This  is  instrumental  in  providing  random  access  into  the  video 
stream  based  on  space  and  time.  Since  the  model  captures  geographic  locations  at  each 
node,  it  is  possible  to  fuse  target  information  into  video-track  information  automatically. 
This  is  accomplished  by  retrieving  all  video  tracks  encompassing  the  geographic  location 
of  the  target.  Tags  are  then  placed  in  the  video  track  identifying  the  offset  location  of 
video  containing  footage  of  the  target. 
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2.5.2  Benefits 


Storing  metadata  at  nodes  and  tagging  video  tracks  with  target  location  makes 
possible  the  retrieval  of  video  based  on  content,  however  limited.  Using  this  model  as  a 
basis,  this  research  will  extend  the  ability  to  retrieve  video  based  not  only  on  the  presence 
of  targets  but  on  how  the  platform  and  sensors  were  used  while  filming  the  targets  as  well 
as  calculated  solar  information.  These  elements  provide  the  ability  to  query  video 
content  with  a  higher  degree  of  granularity. 

2.6  Digital  Versatile  Disk 

With  the  massive  amounts  of  text  and  video  data  being  produced  by  commercial, 
government,  and  military  sectors,  new  and  innovative  ways  are  continuously  being 
sought  to  store  this  information  cheaper,  faster,  and  better.  The  rapid  increase  in  capacity 
and  reduced  price  of  hard  drives  has  answered  the  call  for  stationary  storage  but  portable 
mediums  for  transporting  large  amounts  of  data  remains  a  problem.  The  compact  disk 
(CD)  made  strides  in  this  area  by  introducing  writable  disks  capable  of  storing  up  to  640 
MB  of  data.  While  this  may  be  sufficient  for  many  forms  of  digital  information,  it 
remains  inadequate  for  storing  digitized  video — less  than  two  hours  of  digitized  video  in 
Motion  Picture  Expert  Group  (MPEG)-l  format  and  less  than  an  hour  of  MPEG-2  video 
can  be  stored  on  a  CD.  Recent  advances  in  DVD  technology,  however,  have  introduced 
portable  disks  capable  of  storing  up  to  eight  hours  of  MPEG-2  video.  These  advances 
will  reduce  the  amount  of  disks  needed  to  store  the  mounting  quantities  of  video 
information  becoming  available.  In  the  UAV  community,  this  means  a  single  DVD  can 
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hold  an  entire  eight-hour  mission  reducing  the  number  of  storage  disks  from  6.5  CDs  to  1 
DVD. 

DVD  technology  is  able  to  deliver  much  larger  storage  capacities  by  advancing 
CD  technology  in  two  key  areas.  Firstly,  as  with  CDs,  data  is  recorded  as  a  spiral  trail  of 
pits.  DVDs  obtain  higher  storage  capacities  by  reducing  the  size  of  the  pits  (0.83  jum  to 
0.4  jim)  and  the  distance  between  spiral  layers  (1.6  Jim  to  0.74  |im). 


Figure  4:  Pit  and  Spiral  Dimensions 

Secondly,  DVDs  are  manufactured  with  either  one  or  two  recording  surfaces 

(sides)  and  either  one  or  two  recording  layers  on  each  side  of  the  disk.  Figure  5  shows  a 

representation  of  a  single  sided  disk  with  dual  layers.  The  currently  manufactured 

permutations  of  DVD  configurations  are: 

Single  Sided/Single  Layer  (SS/SL) 

Single  Sided/Double  Layer  (SS/DL) 

Double  Sided/Single  Layer  (DS/SL) 

Double  Sided/Double  Layer  (DS/DL) 
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Layer  1 


Layer  0 


Figure  5:  Cross-Sectional  View  of  DVD  Showing  Two  Layers 

Multiple  recording  layers  coupled  with  smaller  data  pits  and  tighter  spirals  give 
DVDs  the  tremendous  increase  over  CDs  in  storage  capacity.  Table  5  shows  how  DVD 
technology  has  dramatically  increased  the  number  of  hours  of  video  that  can  typically  be 
recorded  on  a  single  disk  (actual  hours  vary  with  the  data  rate  at  which  the  video  was 
recorded  and  number  of  audio  tracks). 


Table  5:  Capacity  of  Video  on  DVD  and  CD  [TAYL098] 


Format 

Sides/Layers 

Hours  of  Video 

DVD 

SS/SL 

2.2  (MPEG-2) 

DVD 

SS/DL 

4.0  (MPEG-2) 

DVD 

DS/SL 

4.4  (MPEG-2) 

DVD 

DS/DL 

8.0  (MPEG-2) 

CD 

SS/SL 

1.3  (MPEG-1) 
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3  METHODOLOGY 


Much  work  is  being  done  to  investigate  methods  that  allow  consumers  of  UAV 
video  products  to  graduate  from  serial  search  and  retrieve  methods  to  random  access 
when  attempting  to  locate  a  video  segment  of  interest.  It  is  proposed  that  by  evaluating 
several  key  sources  of  mission-related  data,  methods  can  be  developed  to  pinpoint 
information-bearing  video  segments.  The  efforts  of  this  research  are  directed  toward 
using  two  sources  of  mission  data  (telemetry  and  target  information)  to  pinpoint  video 
segments  of  interest  and  allow  random  access  into  the  data  stream.  Additionally,  solar 
azimuth  and  altitude  data  will  be  calculated  to  indicate  the  position  of  the  sun  while  the 
video  footage  was  being  recorded.  Since  image  clarity  and  detail  can  be  greatly 
enhanced  with  proper  solar  illumination,  this  data  will  provide  indicators  of  the  potential 
quality  of  the  video  images. 

Random  access  into  the  video  stream  will  be  supported  by  forward-positioning 
data  in  optimized  database  tables  for  rapid  location  and  retrieval  of  video  segments. 
Additionally,  a  user  interface  will  provide  the  capability  to  profile  mission  data  as  well  as 
submit  queries  to  identify  and  retrieve  video  segments  of  interest.  This  chapter  outlines 
characteristics  of  the  data,  the  rationale  for  selecting  metadata,  preparation  and 
calculations  performed  on  the  data,  database  and  interface  design,  and  lastly 
implementation. 
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3.1  Data  Characteristics 


There  is  an  abundance  of  data  associated  with  each  UAV  mission.  Video, 
telemetry,  target,  and  environmental  data  amount  to  many  megabytes  to  gigabytes  of  raw 
data.  It  is  the  goal  of  this  research  to  extract  and  combine  data  elements  that  provide  the 
most  information-bearing  potential,  and  use  them  to  assist  random  access  of  video 
segments.  The  first  stage  in  this  process  involves  understanding  the  data  that  is  available. 
An  introduction  to  the  telemetry  and  target  information  made  available  for  this  research  is 
provided  in  the  following  paragraphs. 

3.1.1  Telemetry  Data 

Telemetry  data  records  continuous  platform  and  sensor  states  for  the  duration  for 
of  a  single  UAV  mission.  Two  sets  of  data  were  provided  by  Air  Force  Research 
Laboratory’s  (AFRL)  Signal  Data  Handling  Branch  (AFRL/IFEC).  One  set,  the  most 
complete  set,  comprised  data  recorded  by  the  Army’s  Hunter  UAV  during  the  All  Service 
Combat  Identification  Evaluation  Team  (ASCIET)  1999  field  evaluations.  Each  record 
in  the  file  contained  the  telemetry  information  associated  with  a  single  frame  in  the  video. 
The  average  file  size  was  13.3  MB  and  was  provided  in  an  American  Standard  Code  for 
Information  Interchange  (ASCII)  format  text  file.  For  the  demonstration  phase  of  this 
research,  data  for  one  mission  was  selected.  This  representative  sample  of  data, 
consisting  of  approximately  39.7  MB  and  covering  almost  five  hours,  was  chosen 
because  it  also  had  accompanying  video  that  would  be  required  in  later  phases  of 
research.  Each  record  contains  26  fields.  The  data  fields  and  their  meaning  are  given 
below. 
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Table  6:  Telemetry  Data  Fields  and  Definitions 


Clock  Time  Hours  (Hr) 

Universal  Time  Coordinated  (UTC)  hour  of  the  mission 

Clock  Time  Minute  (Min) 

UTC  minute  of  the  mission 

Clock  Time  Second  (Sec) 

UTC  second  of  the  mission 

Camera  Center  Look  Point  Latitude 
(CLat) 

Represents  the  geographic  latitude  of  the  center  of  the  frame 
being  recorded  by  the  camera.  Values  are  recorded  in 
decimal-degree  format. 

Camera  Center  Look  Point  Longitude 
(CLon) 

Represents  the  geographic  longitude  of  the  center  of  the 
frame  being  recorded  by  the  camera.  Values  are  recorded  in 
decimal-degree  format. 

Camera  Altitude  (CAlt) 

Represents  the  elevation  above  Mean-Sea-Level  (MSL)  of  the 
location  being  recorded.  Values  are  recorded  in  meters. 

Platform  Latitude  (PLat) 

Current  geographic  latitude  of  the  platform  recorded  in 
decimal-degree  format. 

Platform  Longitude  (PLon) 

Current  geographic  longitude  of  the  platform  recorded  in 
decimal-degree  format. 

Platform  Altitude  (PAlt) 

Current  altitude  of  the  platform  measured  in  meters. 

Platform  Heading  (Heading) 

Direction  the  platform  is  traveling  in  relation  to  North. 

Values  are  0-360  degrees. 

Platform  Roll  (Roll) 

Degree  of  roll  the  vehicle  is  experiencing.  Positive  roll  is  to 
the  right;  negative  is  to  the  left. 

Platform  Pitch  (Pitch) 

Pitch  angle  of  the  vehicle.  Positive  pitch  is  up;  negative  pitch 
is  down. 

Bearing 

Direction  the  camera  was  pointing  in  relation  to  vertical  axis 
of  the  platform.  Values  range  from  0  to  360  degrees. 

Camera  Depression  (CDepr) 

The  extent  to  which  the  camera  was  pointing  down  from  a 
horizontal  position.  0  degrees  is  horizontal  and  90  degrees  is 
directly  below  the  aircraft. 

Zoom  Factor  (Zoom) 

Amount  of  zoom.  Values  range  from  0  (no  zoom)  to  15  (full 
zoom).  If  the  camera  lens  type  (discussed  below)  is  set  to 
doubled,  these  values  are  considered  to  be  doubled. 

Frame  Comer  Latitude 

Geographic  latitude  (in  decimal-degrees)  of  the  recorded 
frame  comer.  Four  values  were  included  to  provide 
coordinates  for  all  frame  comers. 

Frame  Comer  Longitude 

Longitude  (in  decimal-degrees)  of  the  recorded  frame  comer. 
Four  values  were  included  to  provide  coordinates  for  all 
frame  comers. 

Camera  Lens  Type 

Identifies  whether  the  lens  doubler  is  being  used.  The 
doubler  automatically  doubles  the  zoom  factor.  A  value  of  2 
signifies  the  doubler,  6  signifies  a  standard  zoom  factor. 

Camera  Mode 

Identifies  whether  IR  (value  of  1)  video  is  being  gathered  or 
not  (value  of  0). 

Polarity 

Used  when  IR  video  is  being  gathered.  A  value  of  0  means 
that  black  signifies  heat  and  white  signifies  cold.  A  value  of 

1  indicates  the  opposite  effect. 

25 


A  second  set  consisted  of  data  captured  by  Hunter  during  the  Expeditionary  Force 
Experiment  (EFX)  of  1998.  The  selected  samples  from  these  data  sets  contained  the 
same  data  elements  as  the  ASCIET-99  data  set  except  for  the  last  five  fields— Frame 
Comer  Latitude,  Frame  Comer  Longitude,  Camera  Lens  Type,  Camera  Mode,  and 
Polarity.  Unlike  the  ASCIET-99  data  set,  the  EFX-98  data  did  not  contain  a  record  for 
each  frame  of  recorded  video.  In  some  instances,  there  was  as  much  as  15-20 
consecutive  seconds  of  recorded  telemetry  missing.  While  not  optimal,  it  was  determined 
the  data  would  still  provide  enough  coverage  to  meet  the  objective  of  the  experiment:  get 
the  user  as  close  as  possible  to  the  video  segment  of  choice  using  available  telemetry 
information.  This  data  would  also  demonstrate  the  ability  to  accomplish  system 
objectives  in  the  event  portions  of  the  recorded  telemetry  data  were  corrupted  or 
unrecorded. 

Two  EFX-98  data  files  were  chosen  because  they  contained  the  most  complete  set 
of  data  elements  for  which  there  was  accompanying  video.  The  remaining  files  only 
contained  time  and  target  latitude  and  longitude  information;  these  were  unsuitable  for 
this  research.  The  files  were  in  ASCII  format  with  an  average  size  of  approximately 
131.5  KB.  They  consisted  of  data  from  missions  with  duration  of  4-hours  and  30- 
minutes  respectively. 

3.1.2  Target  Data 

Target  data  provides  the  geographic  coordinates  of  targets  projected  to  be 
recorded  during  the  mission — basically  a  flight  plan  or  itinerary.  Target  data  for  EFX-98 
was  provided  by  AFRIVIFEC  in  an  ASCII  format  text  file.  The  file  contained  a  target 
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position  number  and  target  latitude  and  longitude  coordinates  in  degree-minute-decimal 
seconds  (DMS.S)  format. 

Formal  target  data  for  ASCIET-99  was  unavailable.  To  overcome  this  deficiency, 
video  footage  was  observed  and  targets  were  selected  by  correlating  the  time  displayed 
on  the  video  frame  to  telemetry  records  for  the  same  mission.  The  latitude  and  longitude 
values  recorded  in  the  telemetry  file  for  the  correlated  time  were  assigned  to  the  target. 
Additionally,  a  representative  target  designator  was  assigned.  This  was  not  ideal, 
however,  because  one  objective  of  this  research  (Table  1,  Objective2)  involved  using 
target  locations  (as  recorded  in  target  lists)  to  identify  telemetry  records  correlating  to 
video  footage  of  the  target.  Using  notional  data  made  this  impossible.  Final  analysis  of 
the  results  of  this  research,  however,  showed  that  EFX-98  target  and  telemetry  data 
proved  sufficient  to  demonstrate  this  objective. 

3.2  Metadata  Selection 

While  target  and  telemetry  data  serve  as  information  sources  in  their  native 
formats,  they  serve  more  valuably  as  sources  of  indexable  metadata  for  randomly 
accessing  video  segments  of  interest.  Two  approaches  were  considered  in  determining 
the  type  of  metadata  to  extract  or  derive  from  existing  target  and  telemetry  data. 

Metadata  can  be  defined  to  support  a  platform-centric  or  target-centric  view  of  the  data. 

If  a  platform-centric  view  is  chosen,  metadata  should  be  defined  which  provides  insight 
into  how  the  vehicle  was  being  used  during  the  mission.  For  example,  data  items  which 
identify  the  starting  and  ending  time  for  a  segment  where  the  platform  performed  a 
circling  pattern,  or  items  identifying  when  the  camera  was  stationed  off  the  portside  of 
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the  aircraft  would  represent  a  platform-centric  view  of  the  data.  A  platform-centric 
approach  may  prove  most  applicable  in  a  training  context.  For  example,  the  data  could 
be  used  to  determine  if  the  platform  was  navigated  correctly  during  a  circling  pattern  or  if 
the  platform  was  at  the  correct  altitude  during  the  mission.  To  avoid  confusion  later,  it 
should  be  noted  that  although  a  platform-centric  approach  was  not  used  in  this  research  to 
identify  metatdata  items,  the  profile  tool  developed  for  this  research  allows  a  platform¬ 
centric  view  of  the  telemetry  and  selected  metadata.  The  fundamental  difference  is  that  a 
platform-centric  approach  to  metadata  identification  does  not  consider  information 
related  to  what  the  camera  was  viewing;  a  platform-centric  view  of  the  data  merely  refers 
to  evaluating  what  the  camera  was  viewing  from  the  perspective  of  the  platform’s 
location. 

For  this  research,  a  target-centric  approach  to  metadata  identification  was  chosen 
rather  than  a  platform-centric  approach.  Target-centric  data  items  indicate  how  a  specific 
target  was  being  viewed  by  the  platform.  For  example,  which  direction  the  target  was 
being  filmed  from,  or  the  zoom  factor  used  during  filming  are  indications  of  how  the 
target  was  being  viewed.  Targets,  in  this  context,  represent  points  or  areas  of  interest. 
Examples  of  targets  are  1)  objects  at  a  particular  latitude/longitude,  2)  an  area  between 
two  geographic  points,  or  3)  an  area  bounded  by  multiple  geographic  points.  The  target¬ 
centric  approach  was  chosen  because  it  represents  more  closely  the  way  the  data  will  be 
retrieved  in  an  operational  context:  by  areas  or  objects  of  interest  rather  than  how  the 
platform  was  being  used.  While  more  target-centric  data  elements  could  be  chosen,  it  was 
determined  that  the  elements  described  below  provide  the  greatest  insight  into  how  the 
target  was  being  viewed  as  well  as  the  likely  quality  of  the  video  footage.  The  following 
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paragraphs  identify  the  metadata  items  derived  from  telemetry  and  target  data  as  well  as 
calculated  environmental  metadata  items. 

3.2.1  Telemetry 

Telemetry  data  provides  insight  into  the  direction  the  target  was  being  viewed 
from,  the  zoom  factor  used,  and  the  distance  the  platform  was  from  the  target.  Metadata 
describing  each  of  these  was  derived  by  taking  elementary  attribute  values  recorded  in 
the  telemetry  stream  and  combining  or  manipulating  them  to  produce  attributes  of  greater 
semantic  meaning.  Specific  metadata  items  identified  for  this  research  follow. 

Look-Direction.  This  value  is  calculated  based  on  the  look-direction  (CLat, 
CLon)  of  the  camera  and  the  latitude  (PLat)  and  longitude  (PLon)  of  the  platform.  This 
attribute  introduces  the  ability  to  select  video  segments  based  on  the  direction  from  which 
the  target  was  being  viewed.  Values  for  this  data  element  include  eight  azimuth 
regions— N,  NE,  E,  SE,  S,  SW,  W,  and  NW. 


Figure  6:  Look-Direction 
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Zoom.  This  element  provides  an  indication  of  the  probable  level  of  detail  in  the 
image.  Four  categories  of  zoom  values  were  defined  to  reduce  the  number  of  entries  in 
the  database  but  still  provide  sufficient  granularity  for  data  retrieval.  The  categories  are: 


Table  7:  Zoom  Categories 


Low 

recorded  zoom  values  0  to  3.9 

Medium  Low 

recorded  zoom  values  4  to  7.9 

Medium  High 

recorded  zoom  values  8  to  1 1 .9 

High 

recorded  zoom  values  12  to  15 

Look-Distance.  This  is  a  calculated  value  based  on  the  distance  between  the 
camera-center  look-point  and  the  platform  latitude  and  longitude  accounting  for  the 
altitude  of  the  platform.  When  considered  with  zoom,  this  element  provides  an  indication 
of  the  probable  level  of  detail  in  the  image. 


Figure  7:  Look-Distance 

3.2.2  Target 

Target-Designator.  The  target  designator  was  taken  directly  from  the  target  list. 
For  this  research,  notional  designators  were  used.  In  an  operational  setting,  each  target 
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should  be  assigned  a  unique  identifier  that  will  serve  as  the  mission-independent  target 
designator. 

Latitude  and  Longitude.  These  elements  were  also  taken  directly  from  the 
target  list.  The  fields  should  be  in  signed  decimal-degree  format  (e.g.  30.5940,  - 
81 .6385).  Negative  latitudes  represent  coordinates  South  of  the  equator;  negative 
longitudes  represent  coordinates  West  of  the  Prime  Meridian. 

Target-Name.  The  target-name  was  taken  from  the  target  list.  If  a  name  was  not 
supplied,  the  target  name  was  documented  as  Unknown. 

3.2.3  Environmental 

Sun-Angle.  Sun-angle  is  sometimes  referred  to  as  solar  elevation  angle.  It 
identifies  the  angle  at  which  the  sun  was  positioned  during  a  given  time  and  day  of  the 
year.  The  angle  is  determined  by  an  imaginary  line  between  the  target  and  the  sun  and 
the  earth  s  horizon  (Figure  8).  The  value  of  the  angle  will  be  negative  for  periods  during 
which  the  sun  is  below  the  horizon.  This  element  is  calculated  based  on  the  latitude  and 
longitude  of  the  camera  look-point  and  the  time  the  image  was  in  view. 


Figure  8  :  Solar  Altitude  Angle  [GRONB99] 
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Sun  Azimuth.  The  solar  azimuth  angle  is  the  angular  distance  between  due 
South  and  the  position  of  the  sun  at  the  determined  time  (Figure  9).  Positive  values 
represent  positions  East  of  South  while  negative  values  represent  West  of  South 
[GRONB99]. 


Figure  9  :  Solar  Azimuth  Angle  [GRONB99] 

Each  of  the  attributes  mentioned  above  were  chosen  for  their  potential  to  provide 
clues  as  to  the  content  or  quality  of  video  segments.  For  example,  a  user  may  seeks  to 
retrieve  video  segments  which  show  the  greatest  amount  of  detail  over  the  target  area.  To 
increase  the  likelihood  of  retrieving  these  segments,  they  should  select  images  where  the 
sun-direction  and  look-direction  were  the  same.  This  would  indicate  the  target  was  being 
illuminated  on  the  same  side  the  image  was  being  viewed  from  thus  increasing  the 
potential  level  of  detail  that  could  be  expected  in  the  images.  By  using  all  elements 
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cooperatively,  users  can  retrieve  images  with  the  highest  anticipated  image-quality; 
reducing  the  search  time  for  the  “best”  image. 

3.2.4  Other 

Conceptually,  UAV  video  data  consists  of  long  segments  of  recorded  images. 
Each  video  segment  can  be  further  divided  into  smaller  segments  containing  footage  of 
targets.  Finally,  each  target  segment  can  be  further  divided  into  segments  delimited  by  a 
given  action  such  as  a  high  zoom,  an  Easterly  look-direction,  and  the  direction  from 
which  the  sun  was  illuminating  the  target  (Figure  10). 


Video  Segment  1 

Target  Segment  1 

- — • 

Target  Segment  2 

"  - - • 

• - • 

High  Zoom  Med.High  Zoom  Med.  Low  Zoom 

w  — 9  m 

1  km  Look-Distance  1 .5  km  Look-Distance 

w  - # 

North-Easterly  Look-Direction  Easterly  Look-Direction 

—  • - - - • 

South-Easterly  Sun-Direction 

— : - - - • 

Figure  10:  Conceptual  View  of  Video  Stream 
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The  following  metadata  items  were  selected  to  identify  and  delineate  the  various 
segments  of  information. 

MissionMonth.  The  month  the  mission  took  place. 

MissionDay.  The  day  the  mission  took  place. 

VideoSeg.  A  unique  identifier  for  the  video  segment.  For  this  research,  the  file 
name  of  the  Motion  Picture  Expert  Group  (MPEG)-formatted  file  was  used. 

The  next  three  attributes  take  on  different  meanings  based  on  the  context  in 
which  they  are  used.  When  discussing  target  segment,  look-direction,  zoom,  or  look- 
distance  information,  they  represent  values  recorded  in  the  telemetry  record.  When 
referring  to  video  data,  these  attributes  represent  the  displayed  time  on  the  video  frame. 
All  times  are  recorded  in  Universal  Time  Coordinated  (UTC  or  UT). 

StartHr,  StartMin,  StartSec.  These  attributes  represent  the  time  when,  with  a 
high  degree  of  confidence,  a  particular  target  is  anticipated  to  be  in  view  of  the  camera. 
They  are  determined  by  selecting  the  telemetry  record  containing  the  closest  latitude  and 
longitude  values  within  a  specified  distance  (discussed  in  Paragraph  3.4)  of  a  target’s 
latitude  and  longitude.  These  attributes  are  given  the  hour,  minute,  and  second  values 
recorded  in  the  selected  telemetry  record.  Additionally,  these  attributes  mark  the  starting 
time  of  a  video  segment. 

StopHr,  StopMin,  StopSec.  These  attributes  mark  the  stopping  time  of  a 
recorded  segment  of  video. 
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3.3  Database  Design 

The  database  is  implemented  as  a  Microsoft®  Access  relational  database.  Access 
was  chosen  because  it  offers  the  power  of  a  structured  query  language  (SQL)  as  well  as 
support  for  the  data  in  its  native  form— tabular.  Additionally,  since  Microsoft®  is 
prevalent  among  Department  of  Defense  (DOD)  portable  platforms,  Access  offers  a 
maximum  level  of  portability  when  the  system  is  deployed. 

An  entity  relationship  diagram  (ERD)  is  provided  in  Figure  1 1  to  help  the  reader 
conceptualize  the  relationships  that  exist  between  entities  (which  are  later  manifested  as 
tables)  in  the  database.  Brief  discussions  of  the  relationships  follow  the  ERD. 
Additionally,  the  ERD  identifies  the  attributes  contained  within  each  entity  and  their  data 
types.  If  the  entity  participates  as  a  foreign  key  in  a  relationship  with  another  table,  the 
letters  FK  (foreign  key)  appear  beside  the  attribute  name.  Primary  keys  are  made  up  of 
all  attributes  listed  in  the  top  portion  of  each  entity.  Entities  having  rounded  edges 
represent  dependent  entities  whose  primary  key  is  formed  from  attributes  of  one  or  more 
other  entities.  An  example  of  this  is  the  TARGET_SEGMENT  entity  that  takes  attributes 
from  both  the  TELEMETRY  entity  and  the  TARGET  entity  to  form  its  primary  key. 
Square-edged  entities  represent  independent  entities — relying  on  no  other  entities  to  form 
a  primary  key.  The  black  lines  represent  relationships  between  entities.  The  black  dot  is 
placed  on  the  child  side  of  the  parent-child  relationship  between  entities. 


35 


TARGET, SEGMENT  TO  VIDEO  SEGMENT  LOOKUP 


ZOOM 


D Migrator:  String  (FK) 
MiisionMonth:  lntag*r(FK) 
MissionDjy:  Integer  (FK) 
StartHr:  Integer(FK) 
StjrtMln:  Integer(FK) 
StjrtSeo:  lnteger(FKl 
VldeoSeg:  String  (FK) 


LOOK  DIRECTION 


Designator:  String  (FK) 
MIssionMonth:  Integer(FK) 
MissionDjy:  Integer (FK) 
StartHr:  Integer(FK) 
StartMin:  lnt«ger(FK) 
StartSec:  Integer  (FK) 
VldeoSeg:  String  (FK) 


Direction:  String 
SunDireotlon:  String 
SunAltitude:  Integer 


Designator:  String  (FK) 
MisslonMonth:  Integer  (FK) 
MissionDjy:  Integer(FK) 
StirtHr:  Integer  (FK) 
StjrtMln:  Integer  (FK) 
StjrtSeo:  Integer (Fl^ 
VldeoSeg:  String  (FK) 


Zoom:  Integer 


LOOK  DISTANCE 


Designator:  String  (FK) 
MisslonMonth:  integer  (FK) 
MissionDjy:  integer  (FK) 
StartHr  Integer  (FK) 
StartMin:  Integer  (FK) 
StjrtSeo:  Integer(FK) 
VldeoSeg:  String  (FK) 


Distance:  Double 


m>de_up_of 


VIPEO_SEGMENT 


TELEMETRY 


MisslonMonth:  Integer 
MissionDjy:  Integer 
Hr:  Integer 
Min:  Integer 
Sec:  Integer 


CLat:  Double 
CLon:  Double 
CAIt:  Double 
PLat:  Double 
PLon:  Double 
PAIt:  Double 
Heading:  Double 
Roll:  Double 
Pitch:  Double 
Bearing:  Double 
CDepr:  Double 
Zoom:  Double 
Distance:  Double 
Direction:  String 
DlrectionAztmuth:  Integer 
SunAltitude:  Integer 
SunAzimuth:  Integer 
SunDIrection:  String 
ZoomCategory:  Integer 


TARGET  SEGMENT 


MisslonMonth:  Integer 
MissionDjy:  Integer 
StartHr:  Integer 
StartMin:  Integer 
StartSec:  integer 
VldeoSeg:  String 


identifies  start  of. 


Designator:  String  (FK) 
MissionMonth:  Integer  (FK) 
MissionDjy:  Integer(FK) 
StartHr:  Integer  (FK) 
StartMin:  Integer  (FK) 
StartSec:  Integer  (FK) 


TARGET 
Designator:  String  | 

Latitude:  Double  I 
Longitude:  Double  I 
Name:  String  I 


Figure  11:  Entity-Relationship  Diagram 


After  the  database  is  implemented,  TARGET  entries  for  which  there  is  recorded 
TELEMETRY  will  be  included  in  the  TARGET_SEGMENT  entity  (discussed  in  more 
detail  later).  The  TARGET_SEGMENT  entity  participates  in  a  one-to-many  relationship 
with  the  TELEMETRY  entity  as  well  as  the  TARGET  entity.  More  specifically,  the  start 
of  a  target  segment  is  marked  by  attributes  from  a  single  telemetry  record  while  a 
telemetry  record  may  mark  the  start  of  many  target  segments. 

The  TARGET_SEGMENT_TO_VIDEO_SEGMENT_LOOKUP  entity  was 
created  to  relate  target  segments  to  the  video  segments  that  contain  them.  A  target 
segment  may  be  contained  in  only  one  video  segment  while  a  video  segment  may  contain 
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many  target  segments.  Therefore,  TARGET_SEGMENT  participates  in  a  one-to-many 
relationship  with  VIDEO_SEGMENT,  via  the  lookup  table. 

Each  T ARGET_SEGMENT  within  a  given  VEDEO_SEGMENT  has  related 
LOOK_DIRECTION,  LOOK_DISTANCE  and  ZOOM  values.  Each  of  the  former 
entities  participates  in  a  one-to-one  relationship  with  TARGET_SEGMENT  and  a  one-to- 
many  relationship  with  VIDEO_SEGMENT  via  the 

TARGET_SEGMENT_TO_VIDEO_SEGMENT_LOOKUP  entity.  While  it  is  possible 
to  have  multiple  look-directions,  look-distances,  and  zoom  values  for  each  target 
segment,  only  a  single  value  was  assigned  to  minimize  program  complexity  of  the 
implementation  discussed  in  Chapter  4.  It  was  determined  that  a  single  value  was 
sufficient  to  demonstrate  the  objectives  of  this  research;  Chapter  4  bears  this  out. 

3.4  Data  Preparation  and  Database  Population 

Target,  video,  and  telemetry  files,  in  their  native  format,  are  not  sufficient  to  meet 
the  objectives  of  this  research.  During  this  research,  Microsoft®Excel  was  used  as  an 
intermediate  format  during  the  processing  of  target  and  telemetry  files  into  Microsoft® 
Access  Database  tables.  The  video  files  must  be  converted  from  raw  HI-8  video  to 
digitized  MPEG  files.  The  following  steps  identify  the  actions  and  algorithms  used  to 
convert  the  data  and  populate  the  database.  At  this  point,  it  is  assumed  the  Access 
database  has  been  created  and  all  tables  have  been  defined  as  represented  in  the  ERD. 
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Step  1:  Convert  Target  File  to  Excel  Spreadsheet  and  Import  into  Database 


Figure  12:  Target  File  to  Target  Entity  Process  Diagram 

The  target  file  is  converted  into  a  spreadsheet  to  make  it  easier  to  import  into  the 
database.  If  the  file  is  not  in  a  tabular  format,  the  data  must  be  manually  entered  into  the 
spreadsheet.  Once  the  target  file  has  been  opened  or  a  new  file  has  been  created  in  Excel, 
column  headings  are  added.  This  ensures  that  values  are  assigned  to  the  correct  database 
attributes  when  the  spreadsheet  is  imported  into  the  database  in  a  later  action. 

The  target  latitude  and  longitude  values  are  checked  to  ensure  they  are  in  decimal- 
degree  format.  This  is  done  to  ensure  uniformity  of  attribute  formats.  In  some  instances, 
latitudes  and  longitudes  may  be  documented  in  other  formats  (Universal  Transverse 
Mercator  (UTM),  Degrees-Minutes-Second  (DMS)).  In  these  instances,  the  coordinates 
must  first  be  converted  to  decimal-degree  format.  UTM  coordinates  may  be  converted  by 
using  a  shareware  tool  such  as  Corpsconfor  Windows  developed  by  the  Army  Corps  of 
Engineers  [ARMY99].  DMS  can  be  converted  to  decimal-degrees  by  using  the  formula: 

(((Seconds  /  60)  +  Minutes)  /  60)  +  Degrees 
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Degrees-Decimal  Minutes  (DM.M)  may  be  converted  by  using  the  formula: 

( Decimal-Minutes  /  60)  +  Degrees 

Each  target  should  be  assigned  a  unique  identifier.  For  this  research,  if  an 
identifier  was  not  provided  in  the  target  file,  one  was  assigned.  Additionally,  each  target 
should  have  a  name.  Again,  for  this  research,  if  one  was  not  provided  in  the  target  file,  it 
was  assigned  the  value  Unknown.  For  those  instances  when  targets  were  selected  by 
viewing  the  video,  a  notional  designator  and  name  was  assigned. 

The  only  data  items  required  from  the  target  file  are  target  name,  target  latitude 
and  longitude,  and  target  designator.  All  other  items  should  be  deleted  or  omitted  from 
the  spreadsheet. 

The  final  action  involves  importing  the  spreadsheet  into  the  TARGET  table  in  the 
database.  This  is  done  through  the  use  of  an  Access  macro  that  simply  reads  the 
spreadsheet  into  the  table. 
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Step  2:  Convert  Telemetry  File  to  Excel  Spreadsheet  and  Import  into  Database 


Figure  13:  Telemetry  File  to  Telemetry  Table  Process  Diagram 

The  telemetry  file  may  contain  many  unnecessary  records.  One  of  the  files  used 
for  this  research  included  multiple  records  for  each  second  of  video  while  a  second  file 
contained  only  one  record  every  few  seconds  of  video.  The  former  case  introduces 
unnecessary  storage  and  processing  overhead.  Therefore,  an  algorithm  is  necessary  to 
remove  all  extraneous  records  before  opening  the  file  in  Excel  (Figure  14).  For  the 
purposes  of  this  research,  a  UNIX®  Awk  script  was  created  to  ensure  only  one  record  per 
recorded  second  of  telemetry  remained  in  the  file.  The  script  also  appended  the 
MissionMonth  and  MissionDay  attribute  values  to  the  data  stream. 
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Program  ReduceFile(InputFileName  As  String,  MissionMonth  As  Integer, 
MissionDay  As  Integer,  OutputFileName  as  String) 

Open(InputFileName)  For  Reading 
Open(OutputFileName)  For  Writing 
Set  01d_Time  =  99 

While  (GetLine(InputFile)  !=  EOF)  Do 

While(GetLine.Second  =  01d_Time)  Do 
If  (GetLine(InputFile)  =  EOF)  Then  Exit 
End  Do 

Set  01d_Time  =  GetLine.  Second 

WriteLine(OutputFile,  MissionMonth,  MissionDay,  GetLine) 
End  Do 
End  Program 


Figure  14:  Reduce  File  Algorithm 


As  in  Step  1 ,  column  headings  are  added  to  the  spreadsheet  to  assist  importing  the 
spreadsheet  into  the  database.  This  action  is  followed  by  the  calculation  of  the  distance 
between  the  platform  and  the  camera-center  look-point  (CLat,  CLon).  A  geodesic  (Great 
Circle)  distance  computation  is  used  to  calculate  the  distance  between  the  camera-center 
look-point  and  the  coordinates  of  the  vehicle  (PLat,  PLon).  This  provides  the  distance 
along  the  surface  of  the  earth  but  does  not  consider  the  altitude  of  the  vehicle.  The 
algorithm  (Figure  15)  used  to  compute  distance  along  the  surface  of  the  earth  was 
borrowed  from  [KINDR98]  with  modifications  made  to  account  for  the  altitude  of  the 
platform.  The  computational  formula  is  based  upon  the  Haversine  Formula. 
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longitude_difference1  =  longitude  l1  -  longitude21 

latitude_difference1  =  latitude  l1  -  latitude21 

tempi  =  (Sin(latitude_difference/2)A2  +  CosCCLat1)  *  CosCPLat1)  * 
(Sin(longitude_difference/2))A2 

temp2  =  2  *  Atan2(Sqrt(templ),  Sqrt(l-templ))  //  See  Note  2 

earth_distance  =  EARTH_RADIUS  *  temp2  //  EARTH_RADIUS  =  6,367,000  meters 
distance_in_meters  =  Sqrt((earth_distance)A2  +  (PAlt)A2) 

Note  1:  Converted  to  radians  (1  radian  =  IT/ 180) 

Note  2:  Atan2()  calculates  Atan()  and  determines  sign  of  the  output  based  on  the  sign  of  the 
inputs. 


Figure  15:  Distance  Computation  Algorithm 

The  altitude  must  be  considered  to  obtain  the  real  distance  between  the  airborne 
platform  and  the  observed  point.  Real  distance  (c)  is  calculated  by  using  the  Pythagorean 
Theorem  (Figure  16)  with  the  distance  calculated  in  Figure  15  (b)  and  the  altitude  of  the 
vehicle  (h)  as  recorded  in  the  telemetry  record.  Although  not  done  for  this  research, 
Digital  Terrain  Elevation  Data  (DTED)  data  can  be  used  to  provide  more  accuracy  for  the 
altitude  of  the  vehicle  given  the  terrain  at  the  platform  coordinates. 
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Although  equations  exist  for  calculating  exact  distances,  accounting  for  the 
curvature  of  the  earth,  between  target  and  platform,  their  insignificant  precision  gain  does 
not  justify  the  increased  computational  and  programmatic  complexity  induced  by  using 
them.  Figure  17  demonstrates  this  by  showing  the  insignificant  difference  in  the  distance 
computation  using  the  Pythagorean  formula  and  a  formula  for  calculating  the  distance 
between  a  point  on  the  earth  and  an  object  in  the  sky  [PRATT86]. 


Assume  a  platform  whose  coordinates  (in  degrees),  when  projected  on  the  surface  of 
the  earth,  are  30.468N,  -86.8609W  and  a  target  at  coordinates  30.4540N,  -86.8670W. 
The  platform  is  flying  at  an  altitude  of  1 239.3  meters  and  the  distance  between  the 
two  coordinates  (accounting  for  the  curvature  of  the  earth)  is  1661.86  meters. 

Pythagorean  Formula 

a  =  distance  from  target  to  platform 
b  =  altitude  of  platform  -  1239.3  m 

c  =  straight  line  distance  from  target  to  platform  coordinates  =  1661 .86  m 

a  «  b  +  c 

=  2.07  km 


Earth  to  Elevated  Object  Formula 

re  =  radius  of  the  earth  =  6378  km 

r0  =  re  +  altitude  of  the  object  =  6379.2393  km 

Le  =  Latitude  of  point  on  the  earth  =  30.4540  degrees 

le  =  Longitude  of  point  on  the  earth  =  -86.8670  degrees 

L0  =  Latitude  of  object  =  30.468  degrees 

10  =  Longitude  of  object  =  -86.8609 

d  =  distance  from  ground  point  to  platform 


cos(k)  =  cos(Le)cos(L0)cos(l0  -  le)  +  sin(Le)sin(L0) 


d  =  rc  Jl  + 


V  (  -  \ 


) 


-  2 


\r>  ) 


cos 


(A) 


=  2.09  km  (a  delta  of  20  meters  or  about  66  feet) 


Figure  17:  Comparison  of  Formulas  to  Compute  Distance 
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The  direction,  or  heading,  from  the  target  being  observed  to  the  platform  is 
calculated  by  determining  the  angle  (0)  between  the  longitudinal  meridian  of  the  target 
and  the  arc  formed  by  intersecting  the  target  and  the  platform  (d).  The  equation  used  to 
determine  the  angle  is  given  below  [CARVE54]. 


Cos(0)  =  Sin(x21  -  Sinfxll  *  Cosldl 
Sin(d)  *  Cos(xl) 


Figure  18:  Direction  Calculation 
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Since  all  input  values  have  been  derived  during  the  previous  action,  the  direction 
is  easily  derived  using  the  following  algorithm  borrowed  from  [KINDR98]: 


If  (SinCtempl1)  *  Cos(CLat2)  =  0)  Then 

theta  =  0  //  Starting  from  South  Pole 

End  If 

If  (Sin(temp2)  <  0) 3  Then 

theta  =  2  *  PI  -  ACos((Sin(PLat2)  -  Sin(CLat)  *  Cos(temp2))  / 
(Sin(temp2)  *  Cos(CLat))) 

Else 

theta  =  ACos((Sin(PLat)  -  Sin(CLat)  *  Cos(temp2))  / 

(Sin(temp2)  *  Cos(CLat))) 

End  If 

direction  =  theta 

Note  1:  From  Distance  Calculation  Algorithm  in  Figure  15 
Note  2:  Converted  to  radians  (1  radian  =  11/180) 

Note  3:  Since  Westerly  Longitudes  are  given  negative  values.  Otherwise  use 


Figure  19:  Direction  Calculation  Algorithm 

The  next  action  that  is  performed  during  this  step  is  to  calculate  the  solar 
elevation  and  azimuth  angles  for  each  record  in  the  telemetry  spreadsheet.  Each  record  is 
then  updated  with  the  computed  values. 

To  compute  the  elevation  angle,  it  is  necessary  to  know  a)  the  latitude  of  the 
observation  point,  in  this  case,  the  camera-center  look-point,  b)  the  declination  angle  of 
the  sun  at  the  given  time,  and  c)  the  hour  angle  at  the  given  time.  The  reader  is  referred 
to  [GRONB99]  for  a  layman’s  discussion  of  the  meaning  and  derivation  of  declination 
angle  and  hour  angle.  The  elevation  angle  is  calculated  according  to  the  following 
formula: 
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Sin(Al)  -  [  Cos(L)  *  Cos(D)  *  Cos(H)  ]  +  [  Sin(L)  *  Sin(D)  ] 


Where: 

A1  -  Elevation  Angle 
L  =  Latitude  of  observation  point  (CLat) 
D  =  Declination  angle  of  sun 
H  -  Hour  angle 


Figure  20:  Solar  Elevation  Equation  [GRONB99] 


Once  the  elevation  angle  has  been  computed,  the  azimuth  angle  may  be  derived 
and  appended  to  each  telemetry  record  in  the  spreadsheet.  The  following  formula  is  used 
to  calculate  the  values: 


Cos(Az)  -  [  Sin(Al)  *  Sin(L)  -  Sin(D)  ]  /  [  Cos(Al)  *  Cos(L)  ] 


Where: 

Az  =  Solar  azimuth  angle 
A1  =  Elevation  angle 
L  =  Latitude  of  observation  point 
D  =  Declination  angle  of  sun 


Figure  21:  Solar  Azimuth  Equation  [GRONB99] 
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Next,  zoom  values  recorded  in  the  telemetry  record  are  evaluated  and  one  of  the 
zoom  categories  identified  in  Table  7  is  appended  to  the  record.  The  modified 
spreadsheet  is  then  imported  into  the  database  to  populate  the  TELEMETRY  table. 


Step  3:  Populate  T  ARGET_SEGMENT  Table 


Figure  22:  Populate  Target  Segment  Table  Process  Diagram 


This  step  involves  identifying  the  telemetry  records  that  contain  information  about 
video  footage  recorded  over  each  target  in  the  TARGET  table.  Telemetry  records  whose 
camera  coordinates  (latitude  and  longitude)  fall  within  .0003  degrees  of  the  target 
coordinates  (latitude  and  longitude)  are  presumed  to  contain  information  relating  to  that 
target.  The  threshold  of  .0003  degrees  (about  100  feet  at  the  latitude  and  longitude  of  the 
data  provided)  was  chosen  because,  through  experimentation,  it  was  determined  that  this 
was  the  minimum  value  that  would  identify  telemetry  records  for  all  targets  provided — 
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lower  values  did  not  return  telemetry  records  for  all  targets.  A  threshold  this  small  will 
increase  the  probability  the  target  was  in  view  during  recording  since  the  coordinates  of 
the  target  will  be  close  to  the  camera  coordinates.  It  is  understood  that  the  threshold  may 
vary  depending  on  the  altitude  of  the  platform  and  the  zoom  factor  and  look-angle  of  the 
camera.  Future  research  should  consider  a  dynamic  calculation  of  the  threshold  based  on 
the  parameters  mentioned  above. 

Once  all  records  within  the  threshold  are  identified,  the  date  and  time  of  the 
record  with  the  smallest  delta  from  the  target  position  is  selected  to  represent  the  starting 
time  of  the  target  segment.  This  information  is  then  recorded  in  the 
TARGET.SEGMENT  table. 

A  complete  target  segment,  for  the  purposes  of  this  research,  is  designated  as  15 
seconds  prior  to  the  segment  start  time  and  15  seconds  following  the  start  time.  An 
arbitrary  segment  length  was  selected  because  at  this  time  it  is  impossible  to  determine 
when  the  target  is  no  longer  in  view.  This  would  require  knowing  the  target  bounds 
rather  than  a  single  geographic  point  as  is  the  case  with  the  target  data  provided. 
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Step  4:  Populate  VIDEO_SEGMENT  and  TargetSeg_To_VideoSeg_Lookup  Tables 


Figure  23:  Target  and  Video  Segment  to  Lookup  Table  Process  Diagram 

Video  is  typically  supplied  in  analog  format  on  Hi-8  (8mm)  or  VHS  tapes.  It  is 
necessary  to  convert  the  video  stream  into  a  digital  format  since  the  video  viewer  used  for 
this  research  accommodates  digitized  video  only.  This  can  be  accomplished  by  using  a 
software  tool  such  as  SNAZZI™  [DAZZL97]  or  Matrox  PC-VCR  Remote  [MATR099]. 
For  this  research,  SNAZZFM  software  and  hardware  at  the  88th  Communications  Group 
Multimedia  Center  was  used.  Video  segments  of  approximately  70  minutes  in  duration 
were  captured  in  MPEG-1  format  at  a  rate  of  IMB/sec  and  written  to  CD-ROM. 

It  was  neither  necessary  nor  practical  to  digitize  all  the  video  footage  available  for 
this  research.  Therefore,  the  information  contained  in  the  TARGET_SEGMENT  table 
was  used  to  ensure,  at  a  minimum,  segments  containing  footage  of  targets  were  captured. 
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Populating  the  VEDEO_SEGMENT  table  is  done  manually.  The  digitized  video 
file  is  reviewed  to  determine  the  start  and  stop  times  recorded  on  the  video  frames.  The 
database  is  manually  updated  to  reflect  the  segment  name  as  well  as  start  and  stop  times 
and  date. 

The  T ARGET_SEGMENT  and  VIDEO_SEGMENT  records  are  used  to 
determine  which  video  segment  each  target  segment  resides  in.  The  Structured  Query 
Language  (SQL)  code  in  (Figure  24)  is  used  to  determine  this  correlation  and  store  the 
results  in  the  TargetSeg_To_VideoSeg_Lookup  table. 


INSERT  INTO  TargetSeg_To_VideoSeg_Lookup 
SELECT  T.Designator  AS  Designator,  T.MissionMonth  AS  MissionMonth, 
T.MissionDay  AS  MissionDay,  T.StartHr  AS  StartHr,  T.StartMin  AS 
StartMin,  T.StartSec  AS  StartSec,  V.VideoSeg  AS  VideoSeg 
FROM  T ARGET_SEGMENT  AS  T,  VIDEO_SEGMENT  AS  V 
WHERE  (  T.MissionMonth  =  V.MissionMonth 
AND  T.MissionDay  =  V.MissionDay 

AND  ((V.StartHr  <  T.StartHr)  OR  (V.StartHr  =  T.StartHr  AND 
V.StartMin  <  T.StartMin)  OR  (V.StartHr  -  T.StartHr  AND 
V.StartMin  =  T.Startmin  AND  V.StartSec  <=  T.StartSec)) 
AND  ((V.StopHr  >  T.StartHr)  OR  (V.StopHr  =  T.StartHr  AND 
V.StopMin  >  T.StartMin)  OR  (V.StopHr  =  T.StartHr  AND 
V.StopMin  =  T.StartMin  and  V.StopSec  >  T.StartSec))) 
ORDER  BY  V.VideoSeg,  T.Designator,  T.MissionMonth,  T.MissionDay, 
T.StartHr,  T.StartMin,  T.StartSec; 


Figure  24:  SQL  to  Populate  TargetSeg_To_VideoSeg_Lookup  Table 
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Step  5:  Populate  LOOK_DIRECTION,  LOOK_DISTANCE,  and  ZOOM  Tables 


Figure  25:  Look-Direction,  Look-Distance  and  Zoom  Tables  Process  Diagram 

The  TELEMETRY,  VIDEO.SEGMENT,  TARGET_SEGMENT,  and 
TargetSeg_To_VideoSeg_Lookup  tables  are  used  to  populate  the  LOOK_DIRECTION, 
LOOK_DISTANCE,  and  ZOOM  tables.  The  TELEMETRY  and  TARGET_SEGMENT 
tables  are  joined  on  their  time  attributes  (month,  day,  hour,  minute,  second)  to  determine 
the  look-direction  and  look-distance  for  each  record  in  the  TARGET_SEGMENT  table. 
The  LOOK_DIRECTION  and  LOOK_DISTANCE  tables  are  then  populated  with  their 
respective  values. 
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For  zoom  values,  the  time  value  associated  with  each  record  in  the 
TARGET_SEGMENT  table  is  used  to  select  30-seconds  of  recorded  telemetry  records. 
Selecting  telemetry  records  having  a  time  value  up  to  15-seconds  before  and  15-seconds 
after  the  time  recorded  in  the  target  segment  record  identifies  the  30-second  set  of 
telemetry  records.  The  zoom  values  recorded  in  these  records  are  averaged  to  obtain  the 
zoom  value  for  the  ZOOM  table. 

In  addition  to  the  look-direction,  look-distance,  and  zoom  values,  each  table  is 
updated  with  the  target  segment  time  (month,  day,  hour,  minute,  second)  and  the  video 
segment  in  which  the  target  segment  can  be  viewed.  A  sample  query  used  to  populate  the 
LOOK_DIRECTION  and  LOOK_DISTANCE  tables  is  given  in  Figure  26;  Figure  27 
shows  the  query  used  to  populate  the  ZOOM  table. 


SELECT  T.Designator  AS  Designator,  T.MissionMonth  AS  MissionMonth,  T.MissionDay 
AS  MissionDay,  T.StartHr  AS  StartHr,  T.StartMin  AS  StartMin,  T.StartSec  AS 
StartSec,  U.DISTANCE  AS  Distance,  L.VideoSeg  AS  VideoSeg 
FROM  T ARGET_SEGMENT  AS  T,  TELEMETRY  AS  U, 

T argetSeg_To_VideoSeg_LookUp  AS  L,  VIDEO_S EGMENT  AS  V 
WHERE  ( HR  =  T.StartHr  AND  MIN  =  T.StartMin  AND  SEC  =  T.StartSec 
AND  MON  -  T .MissionMonth  AND  DAY  -  T.MissionDay 
AND  T.Designator  =  L.Designator 
AND  T.MissionMonth  *■  L.MissionMonth 
AND  T.MissionDay  =  L.MissionDay 
AND  T.StartHR  =  L.StartHr 
AND  T.StartMin  =  L.StartMin 
AND  T.StartSec  =  L.StartSec 
AND  L.VideoSeg  =  V.VideoSeg 
AND  (((HR*3600)+(MIN*60)+SEC)  >= 

((V.StartHr*3600)+(V.StartMin*60)+V.StartSec)) 

AND  (((HR*3600)+(MIN*60)+SEC)  <= 

((V.StopHr*3600)+(V.StopMin*60)+V.StopSec))) 

ORDER  BY  T.Designator,  T.MissionMonth,  T.MissionDay,  T.StartHr,  T.StartMin, 
T.StartSec,  U.Distance,  L.VideoSeg; 


Figure  26:  Sample  SQL  to  Populate  Look-Direction  and  Look-Distance  Tables 
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INSERT  INTO  Zoom 

SELECT  T.Designator  AS  Designator,  T.MissionMonth  AS  MissionMonth, 
T.MissionDay  AS  MissionDay,  T.StartHr  AS  StartHr,  T.StartMin  AS 
StartMin,  T.StartSec  AS  StartSec,  AVG(U.ZOOMCATEGORY)  AS 
Zoom,  L.VideoSeg  AS  VideoSeg 
FROM  T ARGET_SEGMENT  AS  T,  TELEMETRY  AS  U, 

TargetSeg_To_VideoSeg_LookUp  AS  L,  VIDEO_SEGMENT  AS  V 
WHERE  ( (((HR*3600)+(MIN*60)+SEC)  - 

((T.StartHr*3600)+(T.StartMin*60)+T.StartSec)  >=  -15) 
AND  (((HR*3600)+(MIN*60)+SEC)  - 

((T.StartHr*3600)+(T.StartMin*60)+T.StartSec)  <=  15) 

AND  MON  =  T.MissionMonth  AND  DAY  =  T.MissionDay 

AND  T.Designator  =  L.Designator 

AND  T.MissionMonth  =  L.MissionMonth 

AND  T.MissionDay  =  L.MissionDay 

AND  T.StartHR  =  L.StartHr 

AND  T.StartMin  -  L.StartMin 

AND  T.StartSec  =  L.StartSec 

AND  L.VideoSeg  =  V.VideoSeg 

AND  (((HR*3600)+(MIN*60)+SEC)  >= 

((V.StartHr*3600)+(V.StartMin*60)+V.StartSec)) 

AND  (((HR*3600)+(MIN*60)+SEC)  <= 

((V.StopHr*3600)+(V.StopMin*60)+V.StopSec))) 

GROUP  BY  T.Designator,  T.MissionMonth,  T.MissionDay,  T.StartHr, 
T.StartMin,  T.StartSec,  L.VideoSeg; 


Figure  27:  SQL  Statement  to  Populate  Zoom  Table 


A  finer  level  of  granularity  in  recorded  values  may  be  obtained  by  identifying 
when  each  of  the  attributes  (look-direction,  look-distance,  and  zoom)  changed  value 
rather  than  taking  a  single  representative  value  or  the  average  value  over  the  segment. 
This  was  not  done  for  this  research,  however,  because  the  design  and  coding  effort  was 
not  justified  by  the  research  objectives — locating  and  retrieving  video  segments  based  on 
attributes  of  semantic  value. 
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3.5  Interface  Design 

The  interface  consists  of  two  parts:  a  query  tool  and  a  profiling  tool.  The  query 
tool  is  designed  to  demonstrate  the  ability  to  select  video  segments  based  on  indexed 
attributes  (metadata)  of  high  semantic  meaning.  It  is  target-centric,  meaning  that  all 
queries  are  based  upon  targets  identified  in  the  target  list. 

The  profiling  tool  provides  the  ability  to  graphically  profile  an  entire  mission.  It 
offers  the  ability  to  see  the  mission  from  a  birds-eye  view  while  still  providing  the 
capability  to  observe  information  about  a  single  point.  The  profile  differs  from  the  query 
tool  in  that  it  is  not  restricted  to  retrieving  information  related  to  identified  targets. 
Instead,  it  can  be  used  to  retrieve  information  about  locations  along  the  flight-path  of  the 
platform  that  may  or  may  not  contain  targets.  In  this  respect,  it  provides  access  to  every 
segment  of  recorded  video  rather  than  just  segments  containing  targets. 

This  section  outlines  the  requirements  against  which  the  query  and  profile  tools 
are  designed  and  built.  Also  included  is  a  brief  discussion  of  the  language  and 
application  programmer  interface  (API)  used  to  build  the  tools. 

3.5.1  Requirements 

The  requirements  to  be  observed  during  design  and  implementation  of  the  tools 
are  identified  below.  The  State  Transition  Diagram  (STD)  for  each  tool  identifies  the 
states  the  tools  must  operate  in. 


54 


The  Query  Tool  Shall: 


1 .  Demonstrate  a  capability  for  random  access  into  recorded  video  segments. 

2.  Demonstrate  a  capability  to  access  video  segments  based  on  the  user-selected 
parameters  of  solar  direction,  look-direction,  zoom,  look-distance,  target 
designator,  and  target  name. 

3.  Allow  user-selected  parameters  to  be  chosen  in  any  order  and  any 
combination.  The  user  shall  be  able  to  select  one  or  all  parameters  to  specify 
query  retrieval  characteristics. 

4.  Allow  a  set  of  values  to  be  selected  with  respect  to  solar  direction,  look- 
direction,  and  zoom  parameters.  For  example,  solar  direction  of  South  or 
Southeast. 

5.  Allow  a  range  of  values  to  be  selected  with  respect  to  the  look-distance 
parameter.  For  example,  look-distance  between  3  kilometers  and  6 
kilometers. 


Figure  28:  STD  for  Query  Tool 
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The  Profile  Tool  Shall: 


1.  Demonstrate  a  capability  to  graphically  depict  contiguous  geographic  points 
for  which  telemetry  data  was  recorded  during  a  single  mission. 

2.  Allow  the  user  to  graphically  view  solar  direction,  look-direction,  zoom,  and 
look-distance  values  for  each  point  in  the  profile. 

3.  Allow  the  user  to  select  a  single  point  and  view  the  values  listed  in  Number  2 
above  for  the  selected  point. 

4.  Allow  the  use  of  a  bounding  box  to  zoom  into  an  area  of  interest. 

5.  Allow  the  user  to  view  video  segments  for  any  single  target  or  set  of  points 
displayed  in  the  profile.  The  duration  of  video  based  on  selected  points  will 
be  determined  by  the  time  associated  with  each  point  in  the  set.  The  duration 
of  video  for  a  single  target  will  be  30  seconds  (if  video  exists  for  that 
duration). 


Figure  29:  STD  for  Profile  Tool 
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3.5.2  Implementation 

A  rapid-prototype  approach  is  used  during  construction  of  the  query  and  profile 
tools.  The  support  application  and  language  chosen  should  support  this  methodology. 
Additionally,  the  implementation  language  should  be  highly  portable  across  hardware 
platforms.  For  these  reasons,  Microsoft  Visual  Basic,  Version  6.0,  was  chosen.  Since 
Microsoft  is  the  de  facto  standard  for  DOD  Personal  Computer  (PC)  platforms,  the  tools 
will  meet  the  objective  of  being  highly  portable.  Additionally,  the  Visual  Basic  language 
is  easily  understood  and  offers  the  greatest  opportunity  for  easy  modification  by 
apprentice  or  lay  programmers.  Future  releases  of  the  tools  can  be  modified  quickly  to 
support  changing  user  requirements. 

3.6  Summary 

The  primary  focus  of  this  research  is  to  derive  or  calculate  data  items  from 
mission-related  telemetry  and  target  data  that  can  be  used  as  indexes  to  locate  and 
retrieve  video  segments  of  interest.  The  methodology  of  the  research  consist  of  preparing 
the  source  data,  deriving  or  calculating  metadata  items  to  act  as  indexes  into  the  video 
stream,  preparing  and  populating  the  database,  and  implementing  a  query  and  profile  tool 
to  demonstrate  results.  Chapter  4  includes  screen-shots  of  the  interfaces  and  output  from 
sample  queries  that  demonstrate  the  methodology  discussed  in  this  chapter. 
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4  RESULTS 


This  chapter  presents  results  of  applying  the  methodology  outlined  in  Chapter  3  to 
the  telemetry,  target,  and  video  data  provided  by  AFRL.  It  details  the  steps  taken  to 
realize  the  objectives  of  this  research  as  outlined  in  Table  1.  The  resulting  database  and 
software  application  demonstrate  that  the  telemetry  and  target  data  that  accompanies  a 
UAV  mission,  along  with  calculated  solar  positions,  can  be  used  to  provide  indexes  into 
video  streams  thus  allowing  random  access  and  rapid  location  and  retrieval  of  video 
segments.  Specifically,  this  chapter  covers  the  steps  taken  to  prepare  the  data,  the 
calculation  of  indexable  data  values,  the  construction  and  population  of  the  database,  the 
graphical  user  interfaces  developed  to  view  and  access  the  data,  some  minor  difficulties 
encountered  while  implementing  the  methodology  outlined  in  Chapter  3,  and  finally, 
conclusions. 

4.1  Data  Preparation 

As  outlined  in  Paragraph  3.4,  target  and  telemetry  data  in  their  delivered  formats 
were  not  sufficient  to  fulfill  the  objectives  of  this  research.  EFX-98  target  data  was 
provided  in  an  Excel  spreadsheet  and  a  Word  document.  Target  names  (if  they  were 
available)  and  target  coordinates  were  extracted  from  these  documents  and  manually 
annotated  in  a  new  Excel  spreadsheet.  Target  coordinate  values  were  converted  from 
degrees-minutes-seconds  format  to  decimal-degrees  format  prior  to  being  annotated  in 
the  final  target  spreadsheet.  Figure  30  shows  sample  EFX-98  target  information  and  final 
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Excel  worksheet  used  in  this  research.  No  target  lists  were  available  for  ASCIET-99  so 
notional  targets  were  created  by  viewing  the  video  footage  and  assigning  target 
coordinates  based  on  the  camera-center  look-point  coordinates  in  the  telemetry  record 
corresponding  to  the  time  displayed  on  the  video  frame. 


Position 


Target  Spreadsheet 


WGS-84  Latitude  (N)  WGS-84  Longitude  (W) 


30  33  20.7743 


30  31  50.1972 


30  30  33.9332 


-086  48  48.1874 


-086  47  57.8093 


-086  47  51.0497 


•  •  • 


Range  Coordinates 


UNCLASSIFIED  FOR  EFX-98  USE  ONLY 
EFX-98  EGLIN  AFB  RANGE  SITE  COORDINATES 


BE  Number 

Target  AGL 

Site  1 

303320.9N  0864847.6W 
Connexes  for  command  building 

0467EFX001 

158  ft 

Site  2 

303 150.4N  0864758.1 W 

0467EFX002 

134  ft 

Scud  Reload  Location 


I 


EFX-98  target  data  is  manually 
annotated  in  Excel  spreadsheet 


Designator 

Latitude 

Longitude 

Name 

EFXSitel 

30.5558 

-86.8134 

Connexes  for  command  building 

EFXSite2 

30.5306 

-86.7994 

Scud  Reload  Location 

EFXSite3 

30.5103 

-86.7975 

V  with  1 6  Hulks 

Figure  30:  EFX-98  Target  Information  Flow  into  Target  Spreadsheet 


Telemetry  data  was  converted  from  a  flat-file  format  to  an  Excel  format  to  make 
extracting  and  deriving  indexable  data  items  easier.  In  the  case  of  ASCIET-99  data, 
extraneous  records  had  to  be  eliminated  with  an  UNIX®  Awk  script  before  the  data  was 
imported  into  Excel.  Figure  31  shows  the  data  before  and  after  it  was  reduced  and 
imported  into  and  Excel  spreadsheet  as  well  as  the  Awk  script  used  to  reduce  the  file. 
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Flat  file  containing  telemetry  records 


13,59,42,31.9396, -81.8512,16.1,31.9414, -81.8356, 

13,59,42,31.9396,-81.8512,16.1,31.9414,-81.8356,  •  •  • 

13,59,42,31.9396,-81.8512,16.1,31.9414,-81.8356, 

13,59,42,31.9396,-81.8512,16.1,31.9414,-81.8356, 

X 

input  into 


Output  of  algorithm  is 
input  into  an  Excel 
spreadsheet  prior  to 
calculation  of  additional 
values  (i.e.  look-distance, 
sun-direction,  look- 
direction,  etc.) 


\VMil 

DAY 

im 

SEC 

CLAT 

CLON 

PLON 

3 

13 

59 

42 

31.9396 

-81.8512 

-81.8356 

3 

44 

31.9395 

-81.8511 

16 

31.9422 

-81.8353 

3 

5 

13 

59 

47 

31.9398 

-81.8515 

-81.835 

3 

5 

13 

59 

49 

31.9418 

-81.8501 

-81.8348 

Algorithm  to  remove  extraneous 

{olcL_time  =  $3  #  initialize 

old_time  =  99 
#  Get  next  record 
while  (  getline  >  0) 

{  . 

#  Skip  all  frames  that  have  the  same  time 

#  hack  as  the  one  just  printed  to  file 

while  (  $  3  ==  old_jtime  ) 

{if (getline  !=  1)  exit  } 

#  Got  a  frame  with  the  next  time  hack 

#  Use  the  time  of  the  new  record  as  the 

#time  to  compare  subsequent  frame 

old_time  =  $3 

#  Print  this  record 

print  ARGV [ 2 ]  " , M  ARGV[3]  " ,  "  $0;  temp  -  1 

}  } 


Figure  31:  Flow  of  Telemetry  Data  from  Flat-File  to  Excel  Spreadsheet 
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4.2  Data  Extraction  and  Derivation  Algorithms 

To  complete  the  extraction  and  derivation  of  indexable  data  elements,  the 
algorithms  discussed  in  Chapter  3  (Distance,  Direction,  Solar  Elevation,  Solar  Azimuth, 
and  Zoom  Category)  were  implemented  as  Excel  macros  written  in  Visual  Basic.  Three 
macros  were  developed:  one  to  calculate  distance  and  compass-direction  between 
viewing  area  and  platform,  one  to  calculate  sun  angle  and  azimuth,  and  one  to  determine 
a  general  category  (high,  medium  high,  medium  low,  low)  for  the  zoom  factor  of  the 
camera.  Figure  32  depicts  a  sampling  of  telemetry  records  after  the  algorithms  were 
executed. 


MON 

DAY 

HR 

MIN 

SEC 

CLAT 

CLON 

3 

5 

13 

59 

42 

31.9396 

-81.8512 

3 

5 

13 

59 

44 

31.9395 

-81 .851 1 

3 

5 

13 

59 

47 

31.9398 

-81.8515 

3 

5 

13 

59 

49 

31.9418 

-81.8501 

3 

5 

13 

59 

51 

31.9453 

-81.8498 

ZOOM 

DISTANCE 

DIRECTION 

AZIMUTH 

DIRECTION 

SUN 

ALTITUDE 

SUN 

AZIMUTH 

SUN 

DIRECTION 

ZOOM 

CATEGORY 

1.2 

1.5 

82 

E 

26 

63 

SE 

1 

1 

2 

79 

E 

26 

63 

SE 

1 

1 

2 

78 

E 

26 

63 

SE 

1 

1 

1.5 

82 

E 

26 

63 

SE 

1 

1.6 

1.5 

94 

E 

26 

63 

SE 

1 

Figure  32:  Telemetry  Spreadsheet  with  New  Data  Elements 
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4.3  Database  Creation  and  Table  Population 

The  database  was  implemented  in  Microsoft®  Access  according  to  the  ERD 
described  in  Chapter  3.  The  database  consists  of  eight  tables,  three  queries,  five  macros 
and  two  modules  (Figure  33).  The  queries,  macros,  and  modules  were  developed  to 
assist  in  populating  the  tables  and  are  described  below. 


rwodaks*  i 


[ZOOMj 

VIDEO_5EGMENT 

TELEMETRY 

TargetSeg_To_VldeoSeg_Lookup 

T  ARGET  J5EGMENT 

TARGET 

LOOK_DISTANCE 
LOOK  DIRECTION 


thesis  :  Datab^ 

imm| 


fW*IO| 


Figure  33:  Database  Tables,  Queries,  Macros,  and  Modules 


62 


4.3.1  Queries 


The  following  table  describes  the  SQL  queries  that  were  developed  to  populate 
the  zoom,  target  segment  to  video  segment  lookup,  and  look-distance  tables. 


Table  8  :  Database  Queries 


SelectZoomRecords 

Selects  the  average  zoom-category  value  from  the 
telemetry  records  that  correspond  to  each  target  segment 
recorded  in  the  TARGET_SEGMENT  table.  The  query 
also  populates  the  ZOOM  table  with  the  mission  month, 
mission  day,  start  time  (hour,  minute,  second)  of  the 
related  target  segment,  name  of  the  video  file  containing 
the  target  segment,  target  designator,  and  the  average 
zoom-category  value. 

SelectTargetSeg_To_ 

VideoSegLookUpRecords 

Populates  the  TargetSeg_To_VideoSeg_LookUp  table 
with  attributes  identifying  the  name  of  the  video  file  each 
target  segment  can  be  viewed  from.  The  table  is  updated 
with  mission  month,  mission  day,  start  time  (hour, 
minute,  second)  of  the  target  segment,  name  of  the  video 
file  containing  the  target  segment,  and  the  target 
designator _ 

SelectDistanceRecords 

Selects  the  look-distance  from  the  telemetry  record 
associated  with  each  target  for  which  there  is  a 
TARGET_SEGMENT  record  identified.  The  time 
recorded  in  the  TARGET_SEGMENT  table  is  used  to 
select  the  telemetry  record  containing  the  distance  value. 
The  LOOK_DISTANCE  table  is  updated  with  the  target 
designator,  mission  month,  mission  day,  start  time  (hour, 
minute,  second)  of  the  target  segment,  name  of  the  video 
file  containing  the  target  segment,  and  the  distance 
between  the  platform  and  target. 

4.3.2  Macros 

Macros  were  utilized  for  two  purposes:  to  import  data  from  Excel  spreadsheets 
directly  into  database  tables  and  rebuild  tables  when  new  telemetry  records  were  added  to 
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the  TELEMETRY  table.  The  decision  was  made  to  delete  existing  records  prior  to 
updating  zoom,  target  segment  to  video  segment  lookup,  and  look-distance  tables  to 
eliminate  the  need  for  complex  SQL  or  additional  Visual  Basic  code.  Otherwise,  the 
existence  of  duplicate  records  would  have  to  be  verified  programmatically.  Given  the 
relatively  small  size  of  the  database  used  in  this  research,  deleting  existing  records  prior 
to  rebuilding  tables  did  not  introduce  unacceptable  delays  in  preparing  the  database. 
However,  as  the  amount  of  telemetry  data  increases,  a  more  appropriate  methodology 
should  be  implemented  to  avoid  unacceptable  processing  delays.  Table  9  contains 
descriptions  of  the  macros  used  in  this  research. 


Table  9:  Database  Macros 


GetZoomRecords 

Deletes  existing  records  from  the  ZOOM  table  and 
executes  the  SelectZoomRecords  query 

GetTargetSeg_To_ 

VideoSegLookUpRecords 

Deletes  existing  records  from  the 
TargetSeg_To_VideoSeg_LookUp  table  and  executes  the 
SelectTargetSeg_To_VideoSeg_LookUpRecords  query. 

GetTargetlnfo 

Imports  the  Excel  spreadsheet  containing  the  target 
information  discussed  in  Paragraph  4.1  into  the  TARGET 
table. 

GetMasterTablelnfo 

Imports  the  Excel  spreadsheet  containing  telemetry 
records  into  the  TELEMETRY  table. 

GetDistanceRecords 

Deletes  existing  records  from  the  LOOK_DISTANCE 
table  and  executes  the  SelectDistanceRecords  query. 

4.3.3  Modules 

Visual  Basic  software  modules  were  used  when  a  single  SQL  query  was 
inadequate  to  capture  the  appropriate  information  and  populate  the  database.  These 
circumstances  were  typified  by  a  need  to  make  a  decision  or  transform  the  data  into 
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another  format  prior  to  placing  the  data  into  a  database  table.  The  modules  created  for 
this  research  are  given  in  the  following  table. 


Table  10:  Database  Modules 


SelectTargetSegmentRecords 

Selects  telemetry  records  having  camera-center  look- 
point  coordinates  within  a  specified  distance  (0.0003 
degrees  for  this  research)  of  the  target  coordinates. 

The  recorded  time  from  the  closest  telemetry  record 
is  selected  as  the  median  time  for  the  target  segment 
(the  entire  segment  will  consist  of  15-seconds  of 
video  before  and  after  the  median  time).  The 
TARGET_SEGMENT  table  is  populated  with  the 
target  designator,  mission  month  and  day,  and  the 
selected  time  (hour,  minute,  second). 

SelectLookDirectionRecords 

Selects  the  look-direction  from  the  telemetry  record 
whose  time  matches  the  target-segment  median  time 
as  recorded  in  the  TARGET_SEGMENT  table.  The 
look-direction  is  converted  from  a  numeric  azimuth 
value  (0  to  360)  to  an  8-point  compass  direction  (N, 
NE,  E,  SE,  S,  SW,  W,  and  NW).  The 
LOOK_DIRECTION  table  is  updated  with  mission 
month  and  day,  start  time  (hour,  minute,  second)  of 
the  target  segment,  name  of  the  video  file  containing 
the  target  segment,  target  designator,  look-direction, 
and  the  sun  direction  (azimuth)  and  altitude 
associated  with  the  target  segment. 

4.4  Target-Centric  Query  Tool 

An  interface  was  developed  to  allow  random  access  into  archived  video  streams 
based  on  user-selected  image  characteristics  including  target  name  and  designation, 
camera  zoom  level,  look-distance  between  target  and  platform,  direction  the  target  was 
being  viewed  from,  and  the  direction  from  which  the  sun  was  illuminating  the  target.  The 
following  paragraphs  discuss  how  each  of  these  characteristics  are  used  to  retrieve 
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interesting  target  segments  as  well  as  how  this  information  may  support  mission 
objectives. 

4.4.1  T arget  Look-Direction 

The  target  look-direction  characteristic  provides  a  further  refinement  in  the 
specification  of  video  segments  to  be  retrieved  by  the  query  tool.  Eight  compass 
directions  (N,  NE,  E,  SE,  S,  SW,  W,  NW)  are  available  to  specify  the  look-direction 
(direction  from  which  the  target  was  being  viewed)  that  should  be  reflected  in  the 
retrieved  images.  Figure  34  depicts  a  target  that  was  being  viewed  from  the  West 
meaning  the  platform  was  to  the  West  of  the  target  at  the  time  the  video  segment  was 
recorded. 

Mission  Impact :  This  attribute  is  useful  when  a  specific  view  of  a  target  is  needed. 
For  example,  if  it  is  known  that  a  tank  is  located  on  the  North  side  of  a  building,  then 
video  segments  depicting  a  Northerly  look-direction  will  be  sought.  This  attribute  allows 
the  presence  or  absence  of  these  video  segments  to  be  quickly  ascertained,  and  if  they  do 
exist,  to  quickly  locate  and  retrieve  them  for  viewing.  Currently,  without  reviewing  the 
telemetry  record  corresponding  to  the  video  image,  there  is  no  way  to  determine  the 
direction  from  which  the  target  is  being  viewed.  Direction  can  only  be  calculated  using 
the  geographic  coordinates  of  the  target  and  the  platform.  Without  pre-calculated 
direction  information,  it  becomes  necessary  to  review  all  footage  containing  the  target  in 
hopes  of  finding  the  sought  images. 
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Platform  is  located 
to  the  West  of  the 
target  and  the 
camera  is  looking 
Eastward 


Results  show 
Westerly 
look-direction 
retrieved 


Figure  34:  Query  Results  and  Video  Image  Showing  a  Westerly  Look-Direction 
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4.4.2  Solar  Illumination  Factor 


The  query  tool  can  also  be  used  to  select  video  segments  depicting  a  target  being 
illuminated  by  the  sun  from  a  particular  direction.  While  all  targets  have  a  calculated 
illumination  direction,  this  attribute  is  only  useful  to  describe  targets  recorded  during 
daylight  missions.  Night  missions  are  differentiated  from  daytime  missions  by  either  the 
mission  time  or  the  value  calculated  for  the  sun’s  altitude  angle  at  the  specified  time.  Sun 
angles  of  less  than  zero  denote  night  missions.  By  observing  the  sun’s  altitude  angle,  the 
user  can  determine  if  the  records  returned  by  the  query  correspond  to  daylight  or  night 
mission. 

Mission  Impact:  When  combined  with  look-direction,  this  attribute  can  be  used  to 
select  video  segments  that  offer  the  greatest  degree  of  target  detail.  For  example,  when 
the  sun  is  illuminating  the  target  from  the  same  direction,  or  aft  of  the  platform,  (e.g. 
Easterly  or  Southeasterly  look-direction  and  Easterly  sun-direction)  shadow  obscuring  of 
the  target  is  minimized.  Additionally,  although  not  done  in  this  research,  the  sun’s 
altitude  angle,  as  recorded  in  the  query  results,  provides  a  basis  for  computing  the  length 
of  the  shadow  that  can  be  expected  around  target.  Figure  35  shows  query  results  and 
retrieved  video  images  of  a  target  being  viewed  from  the  South  while  the  sun  was 
illuminating  the  target  from  the  East.  By  observing  the  image,  it  is  clear  that  the  shadows 
produced  by  the  entities  in  the  scene  did  not  obscure  the  target  thus  producing  a  high 
quality  image.  Figure  36  shows  the  types  of  images  that  can  be  avoided  by  using  sun- 
direction  and  look-direction  characteristics.  The  image  on  the  left  shows  a  target 
obscured  by  shadows  as  a  result  of  the  look-direction  be  nearly  opposite  of  sun-direction. 
The  image  on  the  right  shows  a  target  obscured  by  sun-glare  for  the  same  reason. 
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Figure  35:  Query  Results  and  Images  of  a  Southerly  Look  and  Easterly  Sun 


Figure  36:  Video  Images  Obscured  by  Shadowing  and  Sun  Glare 
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4.4.3  Zooms 


The  zoom  image  characteristic  can  be  used  to  further  refine  the  specification  of 
video  segments  to  images  where  the  zoom-category  ranged  from  low  zoom  factors 
(actual  zoom  factors  between  0  and  3)  and  high  zoom  factors  (actual  zoom  factors 
between  12  and  15).  Figure  37  shows  the  query  results  and  a  retrieved  video  image  of  a 
query  to  view  all  target  segments  depicting  high  zoom  images. 

Mission  Impacts :  Selecting  video  segments  where  a  high  zoom  was  used  will 
reveal  the  greatest  amount  of  target  detail  in  the  corresponding  video  images  thus 
enabling  more  precise  identification  of  target  characteristics. 
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Figure  37:  Query  Results  and  Video  Image  of  High  Zoom 
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4.4.4  Look-Distance 


The  look-distance  characteristic  in  the  query  tool  can  be  used  to  further  refine  the 
selection  of  video  segments  to  those  that  were  recorded  when  the  platform  was  within  a 
specified  range  of  the  target.  The  range  starts  at  zero  and  extends  to  the  maximum 
distance  (measured  in  kilometers)  selected  by  the  user.  Targets  that  have  an  associated 
look-distance  of  less  than  or  equal  to  the  maximum  distance  will  be  retrieved. 

Mission  Impact :  As  with  the  other  characteristics  mentioned  previously,  look- 
distance  is  designed  to  give  the  user  the  ability  to  retrieve  the  sharpest,  most  detailed 
images  available  for  analysis  purposes.  Figure  38  reveals  how  target-detail  is  affected  by 
distance  even  at  the  highest  zoom  factor.  The  image  on  the  left  was  recorded  at  a 
distance  of  1  kilometer;  the  image  on  the  right  was  recorded  at  a  distance  of  5  kilometers. 
It  becomes  obvious  that  video  images  depicting  short  look-distances  and  high  zooms 
produce  the  greatest  image  detail. 


Figure  38:  Comparison  of  Different  Look-Distances  at  High  Camera  Zoom 
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4.5  Profile  Tool 


Whereas  the  target-centric  query  tool  provides  the  ability  to  retrieve  video 
segments  containing  images  of  specific  targets,  the  profile  tool  provides  a  graphically 
represented  birds-eye  view  of  the  entire  mission.  Each  of  the  characteristics  described  in 
Section  4.4  are  represented  graphically  in  the  profile  tool  to  quickly  display  information 
about  how  the  platform  and  camera  were  maneuvered  throughout  the  mission. 
Additionally,  with  the  profile  tool,  users  can  retrieve  any  segment  of  video  recorded 
along  the  flight  path  of  the  platform  rather  than  just  video  segments  containing  targets  as 
with  the  query  tool. 

Profiling  mission  data  will  prove  useful  in  both  training  and  operational 
environments.  In  a  training  environment,  a  profile  provides  instructors  or  evaluators  a 
mechanism  for  assessing  whether  the  platform  and  camera  were  used  appropriately  to 
maximize  target  coverage  and  image  quality.  Operationally,  the  graphical  representation 
of  image  characteristics  provides  at-a-glance  indication  of  image  characteristics  (e.g. 
zoom,  look-direction,  look-distance,  and  solar  illumination  direction).  This  will  allow 
rapid  assessment  of  where  target  or  area  coverage  may  have  been  inadequate  and  thus 
require  a  second  look. 

The  profile  tool  builds  the  displayed  image  by  selecting  one  telemetry  record 
representing  each  minute  of  recorded  data.  Sampling  at  this  interval  depicts  ample 
information  to  determine  general  platform  and  camera  usage.  Sampling  at  a  higher  rate 
has  been  shown  to  produce  more  data  points  than  is  necessary  and  causes  the  viewing 
area  to  appear  cluttered. 
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Each  data  point  depicted  in  the  profile  is  actually  a  complex  Active-X  control 
containing  multiple  pieces  of  information  pertaining  to  that  particular  data  point.  This 
information  includes  the  data  point’s  latitude  and  longitude,  the  time  the  data  point  was 
recorded,  and  a  value  for  each  of  the  characteristics  that  can  be  viewed  graphically 
(zoom,  look-distance,  look-direction,  sun-direction).  Storing  the  information  at  the  data 
point  eliminates  the  need  to  query  the  database  prior  to  displaying  graphical  information. 

The  data  used  to  build  the  profile  can  be  viewed  from  a  platform-centric  view  or  a 
camera-centric  view.  The  following  table  describes  how  the  data  in  the  viewing  window 
is  to  be  interpreted  for  each  view. 


Table  11 :  Data  point  Meaning  by  View 


Platform-Centric  View 

Data  points  represent  the  location  of  the  platform  at 
the  given  time.  This  allows  the  user  to  see,  from  the 
perspective  of  the  platform,  how  the  target  area  was 
viewed. 

Camera-Centric  View 

Data  points  represent  the  camera-center  look-point  at 
the  given  time.  This  view  represents  what  the  camera 
was  looking  at  and  the  configuration  of  the  camera  at 
the  time  of  viewing. 

The  following  paragraphs  demonstrate  how  the  profile  tool  is  used  to  represent 
video  image  characteristics.  Each  description  is  followed  by  a  discussion  of  how  the 
information  can  be  used  to  support  military  objectives. 

4.5.1  Zooms 

The  profile  tool  can  be  used  to  view  the  zoom  level  of  the  camera  for  displayed 
data  points.  Figure  39  shows  a  screen  shot  of  the  profile  tool  indicating  points  where 
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high  zooms  were  recorded.  These  points  are  highlighted  by  changing  the  color  of  the 
data  point  to  red. 

Mission  Impact :  Since  high  zooms  produce  video  images  with  the  greatest  amount 
of  detail,  these  segments  will  be  most  often  retrieved.  Therefore,  the  ability  to  rapidly 
locate  and  retrieve  these  segments  is  paramount.  Areas  in  the  profile  depicting  prolonged 
high  zoom  levels  may  indicate  an  area  of  interest  which  requires  a  closer  look  by  analysts 
even  when  a  documented  target  may  not  exist  for  that  location.  These  situations  may 
arise  when  an  object  of  interest  is  observed  while  the  platform  is  in  transit  between 
targets  identified  on  the  flight  plan.  Post  analysis  of  these  images  may  result  in  new 
targets  being  identified.  In  a  training  scenario,  this  graphic  can  be  used  to  determine  if 
the  correct  zoom  factor  was  used  for  the  location  and  objective  in  question. 
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4.5.2  Targets 

Each  target  documented  in  the  database  can  be  displayed  in  the  profile  by  its 
geographic  coordinates.  This  includes  targets  for  which  a  target  segment  was  identified 
as  well  as  those  that  did  not  match  the  criteria  for  identification  of  a  target  segment.  As  a 
reminder,  target  segments  were  only  designated  for  those  targets  whose  coordinates  fell 
within  .0003  degrees  of  the  camera-center  look-point  coordinates  of  existing  telemetry 
records.  For  a  discussion  of  why  this  threshold  was  chosen,  see  Paragraph  3.4,  Step  3. 
Experimentation  revealed  the  .0003-degree  threshold  to  be  a  good  choice  for  the  EFX-98 
telemetry  and  video  used  in  this  research.  This  was  evidenced  by  the  fact  that  all  targets 
identified  in  the  mission  activity  log  that  corresponded  to  the  telemetry  and  video  data 
used  in  this  research  were  identified  and  retrieved. 

Mission  Impact :  Plotting  known  targets  and  data  points  allows  rapid  location  and 
retrieval  of  video  segments  containing  footage  of  targets.  Two  possible  scenarios  exist 
for  determining  when  a  target  may  be  included  in  a  video  segment.  Firstly,  if  the  target  is 
the  primary  focus  of  the  video  segment,  the  target  coordinates  will  fall  within  a 
determined  radius  of  the  camera-center  look-point  coordinates.  The  target-centric  query 
tool  capitalized  on  this  fact  to  retrieve  video  segments  containing  the  target.  Secondly, 
video  footage  may  exist  for  which  the  target  was  displayed  incidentally — meaning  it  was 
not  the  primary  focus  of  the  segment  and  was  not  within  the  determined  radius  of  the 
camera-center  look-point  coordinates.  These  portions  of  video  may  still  prove  useful 
during  analysis  and  should  be  made  available.  The  profile  tool  supports  both  scenarios 
equally  well. 
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By  plotting  data  points  in  a  camera-centric  view  and  noting  the  distance  between 
targets  and  surrounding  points,  all  video  that  may  contain  footage  of  the  target  can  be 
retrieved.  Data  points  falling  close  to  target  indicators  typically  indicate  instances  where 
the  target  was  the  primary  focus  of  the  video.  Those  further  away  correspond  to 
instances  where  the  target  may  appear  incidentally  in  the  video  footage.  Figure  40 
depicts  a  camera-centric  view  of  data  points  and  an  associated  image  revealing  a  target 
that  was  recorded  incidentally.  It  was  determined  to  be  incidental  to  the  segment  because 
the  camera  only  panned  across  the  target  area  without  focusing  on  the  target. 


Figure  40:  Image  Showing  Target  Recorded  Incidentally 
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4.5.3  Look-Direction  and  Solar  Illumination  Direction 


The  look-direction  and  solar  illumination  direction  are  graphically  represented  in 
the  profile  by  colored  lines  corresponding  to  four  possible  compass  directions-N,  E,  S, 
W.  The  look-direction  line  is  colored  blue  to  distinguish  it  from  the  yellow  sun-direction 
line. 

Mission  Impact-.  Like  the  same  characteristics  described  for  the  target-centric 
query  tool  (Paragraph  4.4.1  and  Paragraph  4.4.2),  these  attributes  are  used  together  to 
select  video  segments  with  the  highest  image  detail  and  clarity.  They  can  also  be  used  to 
quickly  identify  if  the  correct  aerial  approach  was  taken  to  the  target  area  to  produce  the 
necessary  images.  For  example,  if  it  is  necessary  to  see  images  of  a  target  as  viewed 
from  an  Easterly  direction,  a  camera-centric  view  of  the  data,  with  accompanying  target 
points  plotted,  will  quickly  reveal  if  the  images  are  available.  Figure  41  shows  the  type 
of  plot  that  reveals  the  characteristics  just  mentioned  and  an  image  from  the  retrieved 
target  video. 
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from  the  East 
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Figure  41:  Look-Direction  and  Solar  Illumination  Direction  Example 
4.5.4  Look-Distance 

Look-distance  is  represented  graphically  by  colored  circles  displayed  under  the 
data  point.  Small  red  circles  represent  a  close  look-distance  (0  to  2  km),  a  slightly  larger 
orange  circle  represents  medium  look-distances  (2  to  4  km),  and  long  look  distances 
(greater  than  4  km)  are  represented  by  large  white  circles. 

Mission  Impact :  The  look-distance  is  useful  in  at  least  two  ways.  As  stated  in  the 
discussion  of  look-distance  as  used  with  the  target-centric  query  tool  (Paragraph  4.4.4), 
this  attribute  is  an  indicator  of  the  likely  image  detail — close  distances  produce  the 
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greatest  amount  of  detail.  However,  this  characteristic  will  also  prove  useful  in  post¬ 
mission  analysis  of  platform  use.  Instances  where  the  platform  was  taken  too  close  to  a 
hot  target  will  be  readily  apparent  in  the  profile. 

Figure  42  shows  data  points  with  look-distance  indicators  revealed.  All  three 
look-distance  graphics  are  depicted. 


Figure  42:  Long,  Medium,  and  Short  Look-Distance  Graphics 


4.5.5  Look-Path 

Look-path  is  the  line-of-sight  path  between  the  camera-center  look-point 
coordinates  and  the  platform  coordinates.  This  feature  provides  a  more  precise 
representation  of  the  actual  look-direction  and  look-distance  between  camera  and 
platform. 
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Mission  Impact :  Displaying  look-paths  provides  an  exact  representation  of  the 
look-direction  and  look-distance  whose  benefits  were  described  in  Paragraph  4.5.3  and 
Paragraph  4.5.4.  Figure  43  shows  a  look-path  for  a  single  data  point.  The  data-point 
represents  the  camera-center  look-point  coordinates  being  viewed  from  a  Westerly 
direction  at  a  medium  distance  (2-4  km)  from  the  platform. 


Figure  43:  Look-Path  for  a  camera-Centric  Data  Point 


4.6  Minor  Problems  Encountered 

The  methodology  proposed  in  Chapter  3  and  the  implementation  described  in  the 
preceding  paragraphs  were  affected  by  a  few  minor  difficulties.  While  these  difficulties 
occurred  very  infrequently  and  did  not  depreciate  the  overall  value  of  this  methodology, 
they  are  presented  here  as  indicators  of  potential  areas  needing  further  research.  The 
difficulties  are  presented  below  by  degree  of  perceived  severity.  They  include  apparent 
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errors  in  telemetry  records  and  missing  telemetry  records,  imprecise  target  coordinates, 
and  the  affects  of  camera  usage  during  the  mission 

4.6.1  Missing  and  Erroneous  Telemetry  Data 

The  supplied  EFX  telemetry  files  were  plagued  by  missing  telemetry  records.  In 
some  instances  the  gap  between  telemetry  records  was  as  high  as  1 1 -seconds.  This  had 
the  minor  affect  of  diminishing  the  accuracy  of  identifying  the  temporal  boundaries  of 
target  segments  and  the  overall  picture  of  how  platform  and  camera  were  used  during  the 
mission.  Although  the  system  proposed  and  implemented  by  this  research  can 
accommodate  missing  records,  more  complete  telemetry  files  will  produce  a  more 
accurate  picture  of  the  mission  and  allow  greater  accuracy  in  locating  targets  identified  in 
the  target  list.  Sparse  data  may  lead  to  targets  not  being  identified  if  the  remaining 
telemetry  records  do  not  record  camera-center  look-points  falling  within  the  .0003-degree 
threshold  as  discussed  in  Paragraph  4.5.2. 

Another  problem,  although  only  occurring  once,  may  indicate  a  more  serious 
system  problem  requiring  further  investigation.  In  the  instance  under  consideration, 
erroneous  camera  coordinates  in  the  telemetry  file  caused  the  identification  of  a  video 
segment  for  a  target,  but  the  target  did  not  appear  in  the  corresponding  video.  The 
problem  arose  because  the  camera  coordinates  of  the  erroneous  record  were  close  to  the 
coordinates  of  the  target.  This  resulted  in  the  telemetry  record  being  selected  as  the  basis 
for  a  target  segment  index  record.  As  a  result,  it  appeared  as  though  a  video  segment 
existed  for  the  target.  This  was  not  the  case,  however,  as  was  discovered  by  viewing  the 
-video  corresponding  to  the  erroneous  target  segment.  In  fact,  the  video  footage  showed 
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that  the  misidentified  target  never  even  entered  the  camera  viewing  area.  Therefore,  it 
was  concluded  that  the  coordinates  in  the  telemetry  record  were  invalid.  This  problem 
can  be  accommodated  by  either  validating  target  segments  against  the  video  stream 
during  population  of  the  database  or  providing  a  capability  to  remove  misidentified  target 
segments  through  the  user  interface. 


The  image  on  the  left  of  Figure  44  shows  a  video  image  corresponding  to  the  time 
recorded  in  the  erroneous  telemetry  record.  Viewing  the  entire  video  sequence  bears  out 
that  this  is  the  same  target  as  contained  in  the  image  on  the  right.  The  desired  target  is 
located  Southeast  of  the  target  revealed  in  these  images. 
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This  is  the  ® 
image  of  the 
verified  target 
at  this  location 


Figure  44:  Images  of  Misidentified  Target  and  Verified  Target 


4.6.2  Lack  of  Target  Boundary  Coordinates 

The  EFX-98  target  data  used  in  this  research  was  characterized  by  each  target 
being  assigned  a  single  geographic  coordinate  (latitude  and  longitude).  There  was  no 
indication  in  the  target  lists  of  the  geographic  area  the  target  covered.  For  example,  a 


82 


building  complex  and  a  tank  both  had  a  single  geographic  coordinate  specifying  their 
respective  locations.  For  small  targets  such  as  vehicles  and  single  buildings,  the  target 
can  be  covered  without  moving  the  camera  too  far  from  the  original  target  coordinates. 

In  the  case  of  a  large  target  area,  however,  the  camera  may  have  to  move  significantly 
from  the  original  target  coordinates.  As  a  result,  by  observing  the  telemetry  stream,  there 
is  no  precise  way  to  determine  when  the  target  is  no  longer  being  recorded.  This  means  a 
precise  target-segment  beginning  and  ending  time  can  not  be  determined. 

Imprecise  target  data  was  accommodated  in  this  research  by  defining  target- 
segments  as  having  duration  of  30-seconds.  The  time  recorded  in  the  telemetry  record 
whose  camera-center  look-point  coordinates  were  closer  to  the  target  coordinates  served 
as  the  center  of  the  30-second  period.  When  video  associated  with  a  target  is  selected 
through  the  query  tool  or  profile  tool,  only  this  30-second  segment  is  initially  presented 
to  the  user.  The  capability  to  backup  or  advance  the  video  stream  was  incorporated  into 
the  application  to  allow  the  user  to  view  all  footage  containing  the  target.  Figure  45 
demonstrates  this  problem  and  the  imprecision  of  the  workaround.  Both  images  reveal 
portions  of  the  target  area  that  were  not  covered  within  the  temporal  bounds  of  the  target 
segment.  The  omitted  footage  is  observed  by  manually  reversing  or  advancing  the  video 
past  the  segment’s  starting  or  ending  points. 

While  more  precision  can  be  obtained  with  prior  knowledge  of  the  exact  bounds 
of  large  targets,  this  knowledge  may  not  always  be  available  or  feasibly  attainable. 
Therefore  the  workaround  described  above  is  necessary  in  lieu  of  using  complex  target- 
recognition  software  to  detect  target  presence  in  the  video. 
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Figure  45:  Lack  of  Target-Segment  Coverage  Due  to  Imprecise  Target  Coordinates 


4.6.3  Effects  of  Camera  Usage 

The  final  minor  problem  involved  the  calculation  of  the  distance  between  the 
camera-center  look-point  coordinates  and  the  platform.  In  several  instances,  the  distance 
calculation  resulted  in  an  extreme  value.  This  situation  resulted  when  the  camera  was 
nearly  horizontal  to  the  platform  causing  the  camera-center  look-point  coordinates  to 
register  as  a  point  on  the  horizon.  This  produced  distance  calculations  in  the  hundreds  of 
kilometers  in  some  instances.  Since  the  display  for  the  profile  tool  is  scaled  using  the 
coordinates  of  the  camera-center  look-points,  the  majority  of  the  data  points  are  often  not 
plotted  uniformly;  rather  they  are  bunched  together  as  in  Figure  46. 

This  problem  can  be  overcome  by  training  camera  operators  to  keep  the  camera 
positioned  towards  the  ground  or  by  fast  processing  of  telemetry  to  eliminate  those 
records  that  exhibit  these  extreme  distance  values.  In  the  latter  case,  no  critical 


information  will  be  lost  since  the  camera  is  not  focused  on  the  ground  and  the  profile 
display  will  look  better. 

An  argument  against  deleting  the  telemetry  records  as  described  above  would  be 
if  proper  camera  and  platform  usage  were  being  evaluated  as  in  a  training  scenario.  In 
these  instances,  deleting  the  records  would  hinder  the  evaluation  process. 
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Figure  46:  Extreme  Distance  Representation  and  Image 


4.7  Conclusions 


The  objectives  of  this  research  as  outlined  in  Table  1  have  been  realized  with  the 
implementation  of  the  methodology  presented  in  Chapter  3.  It  has  been  shown  that 
telemetry  and  target  data  and  calculated  solar  position  can  provide  valuable  indexes 
supporting  random  access  into  video  streams  recorded  during  UAV  missions.  A  query 
tool  such  as  the  one  developed  for  this  research,  supports  target-centric  retrieval  of  video 
segments.  Alternatively,  a  profile  tool  supports  visualization  of  platform  and  camera 
usage  throughout  the  mission.  This  becomes  valuable  when  evaluating  platform  and 
camera  usage  during  the  mission  or  when  selecting  video  segments  which  may  or  may 
not  contain  images  of  a  particular  target. 

The  methodology  presented  in  this  research  can  be  realized  in  a  lightweight, 
portable  application  that  can  be  executed  from  a  laptop  computer  and  a  collection  of  CD- 
ROMs  containing  the  digitized  video  files.  Pre-loading  the  computer  with  indexed 
database  tables  and  the  application  gives  war-fighters  in  the  field  the  ability  to  profile 
past  UAV  missions  and  rapidly  locate  and  retrieve  archived  video  segments.  The 
knowledge  gleaned  from  the  system  can  assist  evaluation  of  past  missions,  retrieval  of 
archived  video  for  comparison  to  recent  video,  or  planning  for  future  missions  by 
identifying  those  areas  or  targets  requiring  a  second  look. 

Chapter  5  will  present  recommendations  for  enhancing  both  the  methodology  and 
the  implementation  presented  in  this  research.  The  recommendations  are  designed  to 
enhance  accuracy  of  the  indexes,  allow  higher  query  granularity,  and  support  the 
portability  of  the  system. 
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5  CONCLUSIONS  AND  RECOMMENDATIONS 


The  focus  of  this  research  is  to  determine  if  mission-related  data,  such  as 
telemetry  and  target  data,  and  calculated  solar  position  can  provide  the  basis  for  indexing 
into  the  video  stream  as  well  as  provide  useful  clues  of  video  content.  Chapter  4 
demonstrates  that  this  is  the  case.  Four  image-describing  characteristics  were  derived 
from  the  mission-related  data  or  calculated  to  demonstrate  this  capability:  look-direction, 
solar  illumination  direction,  look-distance,  and  zoom.  This  chapter  summarizes  the 
benefits  of  this  research  and  provides  recommendations  for  future  research  and 
enhancements  to  the  methodology  and  implementation  to  make  them  more  useful. 

5.1  Benefits 

Benefits  in  both  operational  and  training  support  can  be  realized  by  this  research. 
Operationally,  the  methodology  presented  in  this  research  offers  relief  from  serial 
searching  of  video  to  find  segments  or  images  of  interest.  Analysts  will  now  be  able  to 
access  video  streams  at  any  temporal  point  based  on  target  information,  image 
characteristics,  or  flight  path  coordinates.  Additionally,  they  will  be  able  to  select  video 
segments  of  identified  targets  offering  the  best  images  (images  with  the  greatest  degree  of 
image  detail)  from  a  collection  of  videos  containing  footage  of  the  same  area.  Depending 
on  the  number  of  missions  flown  over  the  same  area,  this  could  dramatically  reduce  the 
time  required  to  locate  and  retrieve  desired  video  images. 
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A  second  benefit  to  be  realized  by  this  research  is  in  the  area  of  mission  analysis 
and  evaluation.  By  reviewing  a  graphical  profile  of  the  mission,  camera  and  platform 
usage  can  be  rapidly  analyzed  and  evaluated.  Insufficient  sensor  coverage  of  a  target 
area,  less  than  optimal  view  of  the  target,  and  approaching  too  close  to  a  hot  target  are 
instances  of  the  types  of  information  which  can  be  quickly  ascertained  through  the  use  of 
the  mission  profile  presented  in  this  research.  Although  only  four  image-describing 
attributes  were  used  in  this  research,  other  attributes  exist  in  the  telemetry  stream  that 
could  prove  useful  to  a  complete  analysis  of  the  platform  and  camera  usage  during  a 
mission.  These  include,  among  others,  camera  depression,  platform  altitude,  and 
heading.  Each  of  these  could  be  incorporated  into  the  profile  to  provide  a  more  complete 
picture  of  sensor  and  platform  configuration  throughout  the  mission. 

5.2  Recommendations 

Although  the  benefits  described  above  are  realizable  in  the  methodology 
presented  in  this  research,  steps  can  be  taken  to  increase  the  accuracy  of  the  indexes, 
introduce  more  indexes,  reduce  the  storage  requirements  of  the  system,  and  enhance  the 
presentation  of  information  to  the  user.  These  steps  include  establishing  a  dynamic 
distance  threshold  for  identifying  target-segments,  using  voice  tapes  to  improve  the 
accuracy  of  target  segment  boundaries  and  introduce  new  indexes,  using  digital  versatile 
disk  (DVD)  technology  to  reduce  video  storage  requirements,  and  using  mosaics  rather 
than  full-motion  video  to  present  video  images  to  users.  Each  of  these  recommendations 
are  presented  below. 
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5.2.1  Dynamic  Distance-Threshold 

For  this  research,  only  one  set  of  target  and  telemetry  data  was  available  for  the 
same  mission:  EFX-98.  The  distance-threshold  (.0003-degrees)  for  determining  if  a 
telemetry  record  contained  information  about  a  target  was  derived  through 
experimentation.  The  lower  bound  was  established  when  all  targets  identified  in  the 
target  list  for  the  mission  in  question  were  associated  with  at  least  one  telemetry  record. 
It  is  hypothesized  that  the  lower  bound  will  be  related  to  the  height  of  the  platform  and 
the  range  from  target  to  platform.  Since  platform  height  did  not  vary  when  the  platform 
was  over  a  target,  this  hypothesis  could  not  be  tested.  It  is  recommended  that  this 
hypothesis  be  researched  further  with  the  intent  of  establishing  a  dynamically 
configurable  lower  bound  on  the  threshold  as  a  function  of  platform  height  and  range  to 
target. 

5.2.2  Mission-Related  Audio  Tapes 

Another  recommendation  that  could  improve  the  accuracy  of  current  indexes  and 
introduce  new  indexes  involves  the  use  of  mission-related  voice  tapes.  These  tapes  may 
offer  useful  information  about  the  exact  location  of  targets  or  other  image-describing 
characteristics.  If  audio  indications  are  made  when  the  camera  enters  and  leaves  a  target 
area,  the  temporal  boundaries  of  target  segments  can  be  more  accurately  defined 
(assuming  the  audio,  video,  and  telemetry  are  temporally  synchronized).  Audio 
recordings  may  also  provide  descriptive  information  about  the  target  itself  such  as 
orientation  (tank  pointed  North),  size  (3-story  building),  or  color  (blue  truck).  This 
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information  provides  attributes  that  assist  in  target  identification  and  should  be  extracted 
and  indexed  to  provide  finer  query  granularity. 

5.2.3  DVD  Technology 

A  third  recommendation  involves  the  storage  medium  and  compression  standard 
used  for  digitized  video  streams.  Currently,  mission  video  is  stored  as  MPEG-1  files  on 
CD-ROMs  with  a  maximum  capacity  of  640  MB.  By  storing  video  files  in  MPEG-2 
format  on  DVDs,  the  storage  capacity  can  be  increased  to  over  4  GB  per  recording  layer 
per  disk  surface  (see  Chapter  2  for  a  discussion  of  how  data  is  stored  on  DVDs).  Four 
gigabytes  of  MPEG-2  compressed  video  equates  to  over  two  hours  of  high-quality  video. 
Therefore,  approximately  eight  hours  of  MPEG-2  video  can  be  stored  on  one  DVD  disk 
as  opposed  to  a  maximum  1 .3  hours  of  MPEG-1  video  on  CD-ROMs.  This  increase  in 
storage  capacity  reduces  the  number  of  disks  that  would  be  deployed  with  a  portable 
system  and  greatly  reduces  the  number  of  disks  required  to  hold  the  digitized  video 
footage  of  all  missions.  Although  the  move  to  DVD  technology  is  advantageous,  care 
must  be  taken  to  ensure  the  DVD  hardware  is  capable  of  reading  CD-ROMs  since  there 
will  likely  be  a  large  amount  of  video-data  stored  on  this  medium— [T A YL098]  points 
out  that  about  half  of  the  first- generation  DVD  drives  can  not  read  the  commonly  used 
CD-R  disks  and  no  CD-ROM  drive  can  read  DVD  disks.  It  is  anticipated  that  when  the 
move  is  made  to  DVDs,  the  most  current  technology  will  be  procured  thus  making  this  a 
non-issue.  Therefore,  it  is  recommended  that  a  migration  path  from  CD-ROM  to  DVD 
technology  be  determined. 
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5.2.4  Mosaics 


One  of  the  challenges  of  full-motion  video  exploitation  lies  in  how  to  present  the 
images  to  the  user  in  such  a  way  as  to  maximize  comprehension  of  the  information.  One 
of  the  effects  that  minimizes  comprehension  is  known  as  the  soda  straw  effect.  This  is 
caused  by  having  only  a  localized  view  of  the  video  information  presented  in  each  frame. 
To  reduce  the  normal  soda  straw  effect,  a  portion  of  video  can  be  viewed  as  a  mosaic  of 
images.  This  birds -eye  view  increases  spatial  comprehension  of  the  data  allowing  more 
accurate  analysis  of  the  information.  Figure  47  demonstrates  this  by  showing  how  more 
information  is  available  to  the  viewer  in  a  single  mosaic  image  than  can  be  relayed  by  a 
single  video  frame. 


A  B 


Figure  47:  Mosaic  Image  (A)  Revealing  More  Information  than  a  Single  Frame  (B) 

Although  there  are  many  benefits  to  using  mosaic  images  rather  than  full-motion 
video,  [PAGE99]  points  out  that  certain  types  of  camera-motion  and  video  characteristics 
hamper  the  mosaic-building  process  used  in  current  software.  Among  these  are  zooming 
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and  tilting  of  the  camera,  dramatic  differences  in  content  between  sequential  frames,  and 
bad  frames  caused  by  poor  camera  exposure  or  environmental  conditions  (sun  glare,  poor 
illumination,  etc.).  However,  this  should  not  deter  efforts  to  capitalize  on  this  technology 
when  the  video  supports  the  mosaic-building  process  as  when  the  camera  is  panning  an 
area.  It  is  recommended  that,  when  the  data  supports  it,  mosaics  be  built  for  video 
segments  containing  targets  identified  in  target  lists.  When  the  user  wish  to  see  a  target 
segment,  the  mosaic  should  be  presented.  Target  segments  for  which  a  mosaic  could  not 
be  built  should  be  returned  as  full-motion  video  as  described  in  this  research. 

5.3  Summary 

Based  on  the  results  presented  in  Chapter  4,  the  methodology  presented  in  this 
research  performs  favorably  to  assist  in  the  rapid  location  and  retrieval  of  video  segments 
of  interest.  Additionally,  the  profiling  of  mission  data  provides  an  informative 
representation  of  how  platform  and  camera  were  utilized  throughout  the  mission.  This 
provides  the  basis  for  the  evaluation  of  platform  and  camera  usage  during  the  mission  as 
well  as  retrieval  of  any  user-specified  segment  of  video.  To  enhance  these  capabilities 
and  to  provide  for  the  conservation  of  resources,  several  recommendations  are  made. 
These  include  introducing  a  dynamic  distance-threshold  for  determining  when  a 
telemetry  record  contains  information  pertaining  to  a  target,  using  voice  tapes  to  gather 
more  information  about  a  target  or  viewing  area,  using  DVD  technology  to  reduce 
storage  requirements,  and,  where  possible,  providing  mosaics  of  target  areas  to  increase 
information  understanding  and  analysis. 
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APPENDIX  A  -  DATA  AND  SOFTWARE  AVAILABILITY 


The  data  and  software  used  in  this  research  is  available  by  contacting  the  AFTT 
School  of  Engineering  and  Logistics  Database  Systems  Research  Point  of  Contact  (POC). 
Currently,  the  Database  Research  POC  is: 


Major  Michael  L.  Talbert,  Ph.D. 

Air  Force  Institute  of  Technology 
WPAFB,  OH  45433-7765 

Email:  michael.talbert@afit.af.mil 

Phone:  DSN  785-6565  ext.  4280  COMM  (937)  255-6565  ext.  4280 
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